Solved
HELP : Floating point operations ? : HELP
Posted on 1997-10-06
I am currently writing a curve fitting component in delphi 1.0 to make our job easier. Now I am multiplying 2 Real numbers and the result obtained for some reason is rounding
to 2 decimal places. It is rounding to .66 when if I trap the variable and convert it to a string on the fly the value
shows up as being .65995 . I need as much accuracy as i can get because I am Curve fitting but because such large numbers are involved I cannot for example multiply the result by 10000 and then round this result and divide by 10000 to reconvert back because it is larger than longint.
It doesn't matter what I set the variable type to ie
REAL,DOUBLE or EXTENDED it still rounds the result. The other curly part is that other operations seem to give me results with 5,6,7 decimal places no worries at all. I know the result should be to at least 5 decimal places as I have also calculated the results in MICROSOFT EXCEL as a check and it gives me the result to 6 decimal places . NO ROUNDING. Can anyone give me any ideas?