We help IT Professionals succeed at work.

OverFlow Error

ISC
ISC asked
on
216 Views
Last Modified: 2010-04-30
I need to store decimal point values in a variable, the range of values is 19.999999 to 0.999999...

The problem is what type do I declare my variable as because....

Integer and Long no good as it rounds decimal places...

Decimal has a range of "For numbers with 28 decimal places, the range is
+/-7.9228162514264337593543950335. The smallest possible non-zero number is 0.0000000000000000000000000001."

Single has a range of "-3.402823E38 to -1.401298E-45 for negative values; 1.401298E-45 to 3.402823E38 for positive values."
 
Variant is the same as "Same range as Double for numerics."  

Thanks Ian

Comment
Watch Question

GrahamSkanRetired
CERTIFIED EXPERT
Top Expert 2012

Commented:
You can store the numbers 19.999999 to 0.999999... in doubles. How many significant digits (how accurate) do you need?
ISC

Author

Commented:
At least 2 significant digits and 6 decimal places...

I tried this but I get an overflow error...


Ian
ISC

Author

Commented:
The calculation causing the problem ss...

   Dim k As Double
       
    k = m_objAccrualRecord.m_intNumberOfDays * 100 * l_dblDailyAccrualTotal

This caused an overflow, however if I change the code to...

  Dim k As Double
       
        k = CDbl(m_objAccrualRecord.m_intNumberOfDays) * CDbl(100) * l_dblDailyAccrualTotal

It works....!
GrahamSkanRetired
CERTIFIED EXPERT
Top Expert 2012

Commented:
There is a buglet in VB. The calculation can start off assuming (for speed) that it is using integers, only to be confounded later on by a number too big to fit into its temporary location.
ISC

Author

Commented:
Thanks I did not know that...!

Commented:
how about


k = l_dblDailyAccrualTotal * 100.0 * m_objAccrualRecord.m_intNumberOfDays

Commented:
or


k = m_objAccrualRecord.m_intNumberOfDays * 100.0 * l_dblDailyAccrualTotal
ISC

Author

Commented:
My solution was to use CDbl( ) in the formula this worked and stopped the overflow error...!

 Dim k As Double
       
        k = CDbl(m_objAccrualRecord.m_intNumberOfDays) * CDbl(100) * l_dblDailyAccrualTotal
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.