Solved

How to control vb.net's fake precision in double values

Posted on 2011-03-11
2
535 Views
Last Modified: 2012-05-11
I have a vb.net program that reads in data values from a text file and store them in double data type. What I discover is that those double values has "added" on some fake precisions. For example, if a value is 0.5 in the text file, it might turn into 0.499999900000000444. This is very annoying as it can accumulate into some serious discrepancies when doing math on these values.

If you just assign 0.5 to a double variable in the code, it will stay as 0.5. I am just not sure how to force VB.net to stop adding on these fake precisions on double values when reading them in from external sources.

I mean, I could just somehow force a precision, but some of the values are truly small. I guess I am asking for a compiler choice or setting somewhere to make VB.net to take in values as is and not to self-indulge in its own interpretation.

Any suggestions?
0
Comment
Question by:TongZ
2 Comments
 
LVL 23

Accepted Solution

by:
wdosanjos earned 25 total points
ID: 35111193
To avoid precision issues, use the Decimal data type instead of Double.

From the documentation:
http://msdn.microsoft.com/en-us/library/xtba3z33%28VS.80%29.aspx

The Decimal data type provides the greatest number of significant digits for a number. It supports up to 29 significant digits and can represent values in excess of 7.9228 x 10^28. It is particularly suitable for calculations, such as financial, that require a large number of digits but cannot tolerate rounding errors.

0
 

Author Closing Comment

by:TongZ
ID: 35112266
Well, I was hoping something more systemic in terms settings etc. I guess a data type change can work. However, Decimal is the slowest of all vb.net data types...
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Write to a printer using vb.net 9 38
Passing data between Forms 3 20
run a stored procedure from vb.net 1 27
jquery tab header text 1 22
This document covers how to connect to SQL Server and browse its contents.  It is meant for those new to Visual Studio and/or working with Microsoft SQL Server.  It is not a guide to building SQL Server database connections in your code.  This is mo…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…

789 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question