Solved

VB.NET Subtraction Problem

Posted on 2006-04-19
Medium Priority
522 Views
Hi,

I'm currenty using VS.NET 2003 .NET 1.1 and am experiencing a very strange problem.

Dim x As Double, y As Double, z As Double

x = 10.2
y = 3
z = x - y

At the end of this z is always  7.199999999999993

Where do all of these decimal places come from? I'm sure there's a very simple answer or I haven't got something set right, but I can't find it.

Jon
0
Question by:HBPROCK
• 2
• 2

LVL 96

Expert Comment

ID: 16490021
It is based on the precision for double.  If you don't need it, don't use it--use singles instead.

Dim x As Single, y As Single, z As Single

x = 10.2
y = 3
z = x - y

Bob
0

Author Comment

ID: 16490205
I see your point Bob, but in certain circumstances (i.e my calculator) I will require a dynamic range of results.

Many more DP may be required, but smaller sums like this will still need to return an accurate result without seemingly erronious variation. Using rounding is not an option...

Could you explain to me where this variation from the math logic comes from? Being a VB6 developer, I've always been used to receiving the 7.2 expected from this calculation.

Many Thanks,
Jon
0

LVL 96

Accepted Solution

Bob Learned earned 400 total points
ID: 16490256
http://msdn2.microsoft.com/en-us/library/x99xtshc(VS.80).aspx

<Quote>
Double precision numbers store an approximation of a real number.

When you work with floating-point numbers, keep in mind that they do not always have a precise representation in memory. This could lead to unexpected results from certain operations...
</Quote>

You only need as much precision from the result that you have with the input:

z = Math.Round(z + .005, 1)

Bob
0

Author Comment

ID: 16490379
This is the information I needed...

Thanks!
Jon
0

Featured Post

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

XML Literals are a great way to handle XML files and the community doesn’t use it as much as it should.  An XML Literal is like a String (http://msdn.microsoft.com/en-us/library/system.string.aspx) Literal, only instead of starting and ending with w…
Introduction When many people think of the WebBrowser (http://msdn.microsoft.com/en-us/library/2te2y1x6%28v=VS.85%29.aspx) control, they immediately think of a control which allows the viewing and navigation of web pages. While this is true, it's a…
Integration Management Part 2
Look below the covers at a subform control , and the form that is inside it. Explore properties and see how easy it is to aggregate, get statistics, and synchronize results for your data. A Microsoft Access subform is used to show relevant calcul…
Suggested Courses
Course of the Month14 days, 23 hours left to enroll