We help IT Professionals succeed at work.

HELP : Floating point operations ? : HELP

boabyte asked
Medium Priority
Last Modified: 2010-04-04
I am currently writing a curve fitting component in delphi 1.0 to make our job easier. Now I am multiplying 2 Real numbers and the result obtained for some reason is rounding
to 2 decimal places. It is rounding to .66 when if I trap the variable and convert it to a string on the fly the value
shows up as being .65995 . I need as much accuracy as i can get because I am Curve fitting but because such large numbers are involved I cannot for example multiply the result by 10000 and then round this result and divide by 10000 to reconvert back because it is larger than longint.
It doesn't matter what I set the variable type to ie
REAL,DOUBLE or EXTENDED it still rounds the result. The other curly part is that other operations seem to give me results with 5,6,7 decimal places no worries at all. I know the result should be to at least 5 decimal places as I have also calculated the results in MICROSOFT EXCEL as a check and it gives me the result to 6 decimal places . NO ROUNDING. Can anyone give me any ideas?
Watch Question

Could you provide a example working code where the problem can be reproduced ? It would helps a lot.


ok the actual code we're using involves matrices so I did a little example to show the weird rounding:

  X,Y,Z : real;
  X := 357.991;
  Y := 357.991;
  Z := X*Y;

Now Z = 128100.28392  in Delphi
    Z = 128100.283921 in Excel

Even more interesting if I use a FloattoStrF to 7 decimal places
(using ffFixed,18,7) I get 128100.2839209 !! And if I convert it
back to the old Z variable I get the same result as the orignal
(128100.28392). You could imagine the affect of this rounding in
matrix operations (least squares method)!!

I write simple code

var e1,e2,e3 : extended;
  e1 := 357.991;
  e2 := 357.991;
  e3 := e1*e2;
  Edit1.Text := FormatFloat( '######.0000000000',e3 );

Now i see result = '128157.556081' not your.
I check this in Excell - the same result.

You must have Pentium with Error or other error in your source.
Try enable in options Pentium save FDIV.

when I use real the result is '128157,55608093
when I use single the result is '128157.554687'
however you can use COMP - this is very big integer with 19-20 places.


      As I have stated I seem to get a good result converting to a string too. The problem is if I do a watch on the e3 variable as the code is working through the result of e3 is 128157.55608.
Have you tried converting Edit1.Text back to a floating point number and then add a breakpoint on the line you put it on then step past it using F7 then have alook at the watch. This is where
it is confusing me. All of my float to string conversions also give me the right answer . The problem is that if I do watches on the real variables they are not displaying the correct results.
Now what I am trying to find out is if the real variables that I am doing the watches on are in fact cutting of decimal places. How is this possible. I have even tried using Int() and Frac() on the real number and adding seperately but still end up with the
same result in the real variable when I put a watch on it.
I need the accuracy of the decimal places to give me a good line fit and if I can't stop simple multiplying errors maybe can't be
helped in Delphi 1.0. Might have to go to Delphi 2.0 or 3.0?
See how you go putting a watch on the real variable Mirek because
I get the same FloatFormat result as you. This is what I have been trying to get across. The strings show the right result and the real number variables show the wrong result!


      Also the number I originally used was 357.911 not 357.991.
This doesn't matter though as I am getting rounding errors everywhere for lots of numbers. Anyway see how you go.
Thanks in Advance.
This is only view in debugging error ? Try use external debuger or other. I just run the Delphi3.....(extended) e3=357.911*357.911=128100.283921
and after float format on screen :
128100,2839210000 so mayby this is the Delphi 1 error.

I just Run Delphi1.........(extended) e3=357.911*357.911=128100.28392
and after float format on screen :
128100,2839210000 so in Delphi3 this works fine, but in delphi1 not.
When you rightclick on watch window then you can edit this watch. In Delphi1 you can set only 11 digits as max for view. This is your error.


      I have already thought of that but really how can I be certain that the figure that the program is using the correct
figure in the variable and not the one shown in the debugging window.Not confident that Delphi1 is using the correct figure.
I try and compare results to Excel but because of the digits being missed off the end the results at the end differ substatially. I have used the least squares method(very simple)
and so is easy to trace. Now if the variables do in fact have the
correct fiqures in them then there should be no discrepancy. I have checked the calcs lots and it is just a matter of inputting
5 x,y points and doing some simple matrix operations(Rows*Columns). But when the final results are output from each program (Excel and Delphi)
The final result is totally different and by a sufficient amount
to warrant me thinking Delphi1 is no good at real number operations. Maybe I should stick to integer operations in Delphi1.
So what do you need ? How can I help you ?

As you see in my previous example Delphi1 stores data internally  in correct value. For extended you have 19-20 digits. This is all digits not digits after ".". When you switch to comp you have 19-20 digits too. You can't have more digits.

If you need help with correcting mistake with your source you must submit this source here.
If you need more digits then you must create own class with multiply, add, devide and more. This class can store number in string as digits. However this can be very slow.


No Mirek the source is fine it is simple mathematics. The thing is I know that the figures delphi is using are the wrong ones now
as I have exhaustively checked the figures by stepping through my code and comparing against results in Excel. Delphi doesn't work the numbers properly and in the end my coefficients are a long wa y out. So I will give you the points for your time unfortunatley I wont be able to use least squares in Delphi1.0 for curve drawing.
PS How can I give you the points unless you lock the question?
Answer ? no problem, but i think so you can use least squares. If you have some time then submit some code mayby the problem can be resolved. If no i have some procedures for working with more digits. I write this some years ago in TurboPascal, but this can be transferred to Delphi.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts


Thanks for your time , I appreciate it muchly.
You don't need my functions ?
If you have ObjectProffesional from TurboPower Software then there is Class for variables stored in BCD. You can use this with own digits number (40-80 or more).

or      try your function with type Comp.
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.