Wrox Programmer Forums Number precision problem
 |
 Excel VBA Discuss using VBA for Excel programming.
 Welcome to the p2p.wrox.com Forums. You are currently viewing the Excel VBA section of the Wrox Programmer to Programmer discussions. This is a community of software programmers and website developers including Wrox book authors and readers. New member registration was closed in 2019. New posts were shut off and the site was archived into this static format as of October 1, 2020. If you require technical support for a Wrox book please contact http://hub.wiley.com

June 28th, 2004, 11:13 AM
 adriant42 Authorized User Join Date: Jun 2004 Posts: 11 Thanks: 0 Thanked 0 Times in 0 Posts
Number precision problem

Hi:

I have encountered a precision error. Hopefully, someone could answer this.

In a class (clsAssumptions), I have following:
---------------------------------------------------------
Public Function RefundEarnedPremiumSchd(ByRef arrSchedule() As Single)
Dim i As Integer

'Return array of earned premium schedule to the calling procedure
arrSchedule(1) = 0.1
arrSchedule(2) = 0.2
arrSchedule(3) = 0.175
arrSchedule(4) = 0.135
arrSchedule(5) = 0.11
arrSchedule(6) = 0.09
arrSchedule(7) = 0.07
arrSchedule(8) = 0.05
arrSchedule(9) = 0.035
arrSchedule(10) = 0.035

For i = 11 To 30
arrSchedule(i) = 0
Next i

End Function
---------------------------------------------------------

In a module, I have a calling procedure to populate cells with above numbers:

---------------------------------------------------------
Dim Assumptions As clsAssumptions
Dim i As Integer
Set Assumptions = New clsAssumptions

Range("A1")(1, 1).Activate
For i = 1 To 30
ActiveCell.Next.Activate
Next i

Set Assumptions = Nothing

End Sub
---------------------------------------------------------

When I go to the sheet, the results are:

0.100000001490116 (input is 0.1)
0.200000002980232 (input is 0.2)
0.174999997019768 (input is 0.175)
0.135000005364418
0.109999999403954
0.090000003576279
0.070000000298023
0.050000000745058
0.035000000149012
0.035000000149012

QUESTIONs: Where are extra decimals coming from? How can I eliminate extra decimals? I want 0.1, not 0.100000001490116.

June 28th, 2004, 12:41 PM
 Birger Friend of Wrox Join Date: Jun 2003 Posts: 150 Thanks: 0 Thanked 0 Times in 0 Posts

arrSchedule

to Variant types

According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time.
Have a try!

June 28th, 2004, 01:00 PM
 adriant42 Authorized User Join Date: Jun 2004 Posts: 11 Thanks: 0 Thanked 0 Times in 0 Posts

Hi:

The CDec function solves the problem. Thanks!!!

Quote:
 quote:Originally posted by Birger  Try to change your variables: arrRefundEarnedPremium and arrSchedule to Variant types Then change your line in your sub to this: ActiveCell.Value = CDec(arrRefundEarnedPremium(i)) According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time. Have a try!

 Similar Threads Thread Thread Starter Forum Replies Last Post Scale and Precision in number Data type code_lover Oracle 1 April 9th, 2009 04:56 PM possible loss of precision hobby Java Basics 7 September 19th, 2008 09:41 PM Float precision nulogix C++ Programming 1 November 10th, 2005 09:53 PM How to control the decimal point precision johnsonlim026 ASP.NET 1.0 and 1.1 Basics 0 August 7th, 2005 11:58 PM Length vs precision in decimal datatype rvw SQL Server 2000 2 January 5th, 2004 12:36 PM