Wrox Programmer Forums

Need to download code?

View our list of code downloads.

Go Back   Wrox Programmer Forums > Microsoft Office > Excel VBA > Excel VBA
Password Reminder
Register
| FAQ | Members List | Search | Today's Posts | Mark Forums Read
Excel VBA Discuss using VBA for Excel programming.
Welcome to the p2p.wrox.com Forums.

You are currently viewing the Excel VBA section of the Wrox Programmer to Programmer discussions. This is a community of tens of thousands of software programmers and website developers including Wrox book authors and readers. As a guest, you can read any forum posting. By joining today you can post your own programming questions, respond to other developers’ questions, and eliminate the ads that are displayed to guests. Registration is fast, simple and absolutely free .
DRM-free e-books 300x50
Reply
 
Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old June 28th, 2004, 11:13 AM
Authorized User
 
Join Date: Jun 2004
Location: , , USA.
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
Default Number precision problem

Hi:


I have encountered a precision error. Hopefully, someone could answer this.


In a class (clsAssumptions), I have following:
---------------------------------------------------------
Public Function RefundEarnedPremiumSchd(ByRef arrSchedule() As Single)
  Dim i As Integer

'Return array of earned premium schedule to the calling procedure
  arrSchedule(1) = 0.1
  arrSchedule(2) = 0.2
  arrSchedule(3) = 0.175
  arrSchedule(4) = 0.135
  arrSchedule(5) = 0.11
  arrSchedule(6) = 0.09
  arrSchedule(7) = 0.07
  arrSchedule(8) = 0.05
  arrSchedule(9) = 0.035
  arrSchedule(10) = 0.035

  For i = 11 To 30
    arrSchedule(i) = 0
  Next i

End Function
---------------------------------------------------------

In a module, I have a calling procedure to populate cells with above numbers:

---------------------------------------------------------
Public Sub popRefundEarnedPremiumSchd()
  Dim Assumptions As clsAssumptions
  Dim arrRefundEarnedPremium() As Single
  Dim i As Integer
   Set Assumptions = New clsAssumptions

  ReDim arrRefundEarnedPremium(1 To 30)
  Assumptions.RefundEarnedPremiumSchd arrRefundEarnedPremium()
  Range("A1")(1, 1).Activate
  For i = 1 To 30
    ActiveCell.Value = arrRefundEarnedPremium(i)
    ActiveCell.Next.Activate
  Next i

  Set Assumptions = Nothing

End Sub
---------------------------------------------------------

When I go to the sheet, the results are:

0.100000001490116 (input is 0.1)
0.200000002980232 (input is 0.2)
0.174999997019768 (input is 0.175)
0.135000005364418
0.109999999403954
0.090000003576279
0.070000000298023
0.050000000745058
0.035000000149012
0.035000000149012


QUESTIONs: Where are extra decimals coming from? How can I eliminate extra decimals? I want 0.1, not 0.100000001490116.
Reply With Quote
  #2 (permalink)  
Old June 28th, 2004, 12:41 PM
Friend of Wrox
 
Join Date: Jun 2003
Location: , , Denmark.
Posts: 150
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Try to change your variables:

arrRefundEarnedPremium and
arrSchedule

to Variant types

Then change your line in your sub to this:

ActiveCell.Value = CDec(arrRefundEarnedPremium(i))

According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time.
Have a try!
Reply With Quote
  #3 (permalink)  
Old June 28th, 2004, 01:00 PM
Authorized User
 
Join Date: Jun 2004
Location: , , USA.
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Hi:


The CDec function solves the problem. Thanks!!!



Adrian T



Quote:
quote:Originally posted by Birger
 Try to change your variables:

arrRefundEarnedPremium and
arrSchedule

to Variant types

Then change your line in your sub to this:

ActiveCell.Value = CDec(arrRefundEarnedPremium(i))

According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time.
Have a try!
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Scale and Precision in number Data type code_lover Oracle 1 April 9th, 2009 04:56 PM
possible loss of precision hobby Java Basics 7 September 19th, 2008 09:41 PM
Float precision nulogix C++ Programming 1 November 10th, 2005 09:53 PM
How to control the decimal point precision johnsonlim026 ASP.NET 1.0 and 1.1 Basics 0 August 7th, 2005 11:58 PM
Length vs precision in decimal datatype rvw SQL Server 2000 2 January 5th, 2004 12:36 PM



All times are GMT -4. The time now is 03:25 PM.


Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
© 2013 John Wiley & Sons, Inc.