Solved

How to control vb.net's fake precision in double values

Posted on 2011-03-11
2
547 Views
Last Modified: 2012-05-11
I have a vb.net program that reads in data values from a text file and store them in double data type. What I discover is that those double values has "added" on some fake precisions. For example, if a value is 0.5 in the text file, it might turn into 0.499999900000000444. This is very annoying as it can accumulate into some serious discrepancies when doing math on these values.

If you just assign 0.5 to a double variable in the code, it will stay as 0.5. I am just not sure how to force VB.net to stop adding on these fake precisions on double values when reading them in from external sources.

I mean, I could just somehow force a precision, but some of the values are truly small. I guess I am asking for a compiler choice or setting somewhere to make VB.net to take in values as is and not to self-indulge in its own interpretation.

Any suggestions?
0
Comment
Question by:TongZ
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 23

Accepted Solution

by:
wdosanjos earned 25 total points
ID: 35111193
To avoid precision issues, use the Decimal data type instead of Double.

From the documentation:
http://msdn.microsoft.com/en-us/library/xtba3z33%28VS.80%29.aspx

The Decimal data type provides the greatest number of significant digits for a number. It supports up to 29 significant digits and can represent values in excess of 7.9228 x 10^28. It is particularly suitable for calculations, such as financial, that require a large number of digits but cannot tolerate rounding errors.

0
 

Author Closing Comment

by:TongZ
ID: 35112266
Well, I was hoping something more systemic in terms settings etc. I guess a data type change can work. However, Decimal is the slowest of all vb.net data types...
0

Featured Post

Instantly Create Instructional Tutorials

Contextual Guidance at the moment of need helps your employees adopt to new software or processes instantly. Boost knowledge retention and employee engagement step-by-step with one easy solution.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VB.net/VSTO Excel Add-in 2 36
SQL Server Serialization error 8 43
vb.net dbnull syntax 1 51
Difference between Highcharts and Mapbox 10 44
More often than not, we developers are confronted with a need: a need to make some kind of magic happen via code. Whether it is for a client, for the boss, or for our own personal projects, the need must be satisfied. Most of the time, the Framework…
Parsing a CSV file is a task that we are confronted with regularly, and although there are a vast number of means to do this, as a newbie, the field can be confusing and the tools can seem complex. A simple solution to parsing a customized CSV fi…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…

737 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question