?
Solved

How to control vb.net's fake precision in double values

Posted on 2011-03-11
2
Medium Priority
?
579 Views
Last Modified: 2012-05-11
I have a vb.net program that reads in data values from a text file and store them in double data type. What I discover is that those double values has "added" on some fake precisions. For example, if a value is 0.5 in the text file, it might turn into 0.499999900000000444. This is very annoying as it can accumulate into some serious discrepancies when doing math on these values.

If you just assign 0.5 to a double variable in the code, it will stay as 0.5. I am just not sure how to force VB.net to stop adding on these fake precisions on double values when reading them in from external sources.

I mean, I could just somehow force a precision, but some of the values are truly small. I guess I am asking for a compiler choice or setting somewhere to make VB.net to take in values as is and not to self-indulge in its own interpretation.

Any suggestions?
0
Comment
Question by:TongZ
2 Comments
 
LVL 23

Accepted Solution

by:
wdosanjos earned 75 total points
ID: 35111193
To avoid precision issues, use the Decimal data type instead of Double.

From the documentation:
http://msdn.microsoft.com/en-us/library/xtba3z33%28VS.80%29.aspx

The Decimal data type provides the greatest number of significant digits for a number. It supports up to 29 significant digits and can represent values in excess of 7.9228 x 10^28. It is particularly suitable for calculations, such as financial, that require a large number of digits but cannot tolerate rounding errors.

0
 

Author Closing Comment

by:TongZ
ID: 35112266
Well, I was hoping something more systemic in terms settings etc. I guess a data type change can work. However, Decimal is the slowest of all vb.net data types...
0

Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
This video shows how to quickly and easily deploy an email signature for all users in Office 365 and prevent it from being added to replies and forwards. (the resulting signature is applied on the server level in Exchange Online) The email signat…
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
Suggested Courses

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question