Link to home
Create AccountLog in
Avatar of Tom Farrar
Tom FarrarFlag for United States of America

asked on

Simple Excel Calculation..

Why doesn't one of these formulas in attached spreadsheet columns B,C,D,E give me an answer of $.75?  Not sure I understand why the $.750000000000002...  What am I missing?  Thanks.
EE.xlsx
Avatar of Sean
Sean
Flag of United States of America image

Adjust your cell formatting to only display the .75
Avatar of Tom Farrar

ASKER

It is not just formatting.  I need the number to be exactly $.75
Avatar of Norie
Norie

Try =ROUND(B2-B3,2) etc.
Its all to do with Excel's built in level of accuracy; It will only calculate to 15 significant places.

However, that doesn't help explain why two hard-coded (as opposed to calculated) numbers that are only 3/4 significant figures (2 before decimal and 2 after decimal) subsequently create a 15th significant figure, the 2 at the very end.
Try:
=ROUNDDOWN(B2-B3,2)

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of Norie
Norie

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Why doesn't one of these formulas in attached spreadsheet columns B,C,D,E give me an answer of $.75?  Not sure I understand why the $.750000000000002...  What am I missing?  Thanks.

select the cells and use the number format Currency―problem solved
EE_currrency.xlsx
The inaccuracy is caused by underlying data type which is "double" for the standard number. Double uses binary representation in 64 bits split between mantissa and exponent and the important thing is not all decimal numbers do have exact conversion to binary representation. Some (for us) simple numbers containing fractional part are converted to periodic binary number so certain rounding must happen.

Currency data type uses big integer (64 bits) with emulated 4 decimals obviously. So the accuracy is ensured if the 4 decimal places are enough. I am not sure whether Excel uses currency data type or just rounds the values.

Many programming languages and database engines (and some CPUs) use decimal data type which calculates and stores the decimal representation of the number without any conversion. Such data type cannot loose accuracy the same way as double. Of course calculations are much slower when used CPU does not support decimal data type.

Update: Sorry, the first paragraph was explained by Norie already.
There were good thoughts by all, but Norie was first and most concise.  Thank you all.