Wrox Programmer Forums
Go Back   Wrox Programmer Forums > Microsoft Office > Excel VBA > Excel VBA
| Search | Today's Posts | Mark Forums Read
Excel VBA Discuss using VBA for Excel programming.
Welcome to the p2p.wrox.com Forums.

You are currently viewing the Excel VBA section of the Wrox Programmer to Programmer discussions. This is a community of software programmers and website developers including Wrox book authors and readers. New member registration was closed in 2019. New posts were shut off and the site was archived into this static format as of October 1, 2020. If you require technical support for a Wrox book please contact http://hub.wiley.com
 
Old June 28th, 2004, 11:13 AM
Authorized User
 
Join Date: Jun 2004
Location: , , USA.
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
Default Number precision problem

Hi:


I have encountered a precision error. Hopefully, someone could answer this.


In a class (clsAssumptions), I have following:
---------------------------------------------------------
Public Function RefundEarnedPremiumSchd(ByRef arrSchedule() As Single)
  Dim i As Integer

'Return array of earned premium schedule to the calling procedure
  arrSchedule(1) = 0.1
  arrSchedule(2) = 0.2
  arrSchedule(3) = 0.175
  arrSchedule(4) = 0.135
  arrSchedule(5) = 0.11
  arrSchedule(6) = 0.09
  arrSchedule(7) = 0.07
  arrSchedule(8) = 0.05
  arrSchedule(9) = 0.035
  arrSchedule(10) = 0.035

  For i = 11 To 30
    arrSchedule(i) = 0
  Next i

End Function
---------------------------------------------------------

In a module, I have a calling procedure to populate cells with above numbers:

---------------------------------------------------------
Public Sub popRefundEarnedPremiumSchd()
  Dim Assumptions As clsAssumptions
  Dim arrRefundEarnedPremium() As Single
  Dim i As Integer
   Set Assumptions = New clsAssumptions

  ReDim arrRefundEarnedPremium(1 To 30)
  Assumptions.RefundEarnedPremiumSchd arrRefundEarnedPremium()
  Range("A1")(1, 1).Activate
  For i = 1 To 30
    ActiveCell.Value = arrRefundEarnedPremium(i)
    ActiveCell.Next.Activate
  Next i

  Set Assumptions = Nothing

End Sub
---------------------------------------------------------

When I go to the sheet, the results are:

0.100000001490116 (input is 0.1)
0.200000002980232 (input is 0.2)
0.174999997019768 (input is 0.175)
0.135000005364418
0.109999999403954
0.090000003576279
0.070000000298023
0.050000000745058
0.035000000149012
0.035000000149012


QUESTIONs: Where are extra decimals coming from? How can I eliminate extra decimals? I want 0.1, not 0.100000001490116.
 
Old June 28th, 2004, 12:41 PM
Friend of Wrox
 
Join Date: Jun 2003
Location: , , Denmark.
Posts: 150
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Try to change your variables:

arrRefundEarnedPremium and
arrSchedule

to Variant types

Then change your line in your sub to this:

ActiveCell.Value = CDec(arrRefundEarnedPremium(i))

According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time.
Have a try!
 
Old June 28th, 2004, 01:00 PM
Authorized User
 
Join Date: Jun 2004
Location: , , USA.
Posts: 11
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Hi:


The CDec function solves the problem. Thanks!!!



Adrian T



Quote:
quote:Originally posted by Birger
 Try to change your variables:

arrRefundEarnedPremium and
arrSchedule

to Variant types

Then change your line in your sub to this:

ActiveCell.Value = CDec(arrRefundEarnedPremium(i))

According to what I've found, the problem arise when using floating point variables. However, there seem to be numerous problems regarding this issue, so no guarantee it should work every time.
Have a try!




Similar Threads
Thread Thread Starter Forum Replies Last Post
Scale and Precision in number Data type code_lover Oracle 1 April 9th, 2009 04:56 PM
possible loss of precision hobby Java Basics 7 September 19th, 2008 09:41 PM
Float precision nulogix C++ Programming 1 November 10th, 2005 09:53 PM
How to control the decimal point precision johnsonlim026 ASP.NET 1.0 and 1.1 Basics 0 August 7th, 2005 11:58 PM
Length vs precision in decimal datatype rvw SQL Server 2000 2 January 5th, 2004 12:36 PM





Powered by vBulletin®
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) 2020 John Wiley & Sons, Inc.