• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 64
  • Last Modified:

Simple Excel Calculation..

Why doesn't one of these formulas in attached spreadsheet columns B,C,D,E give me an answer of $.75?  Not sure I understand why the $.750000000000002...  What am I missing?  Thanks.
EE.xlsx
0
tomfarrar
Asked:
tomfarrar
1 Solution
 
SeanSystem EngineerCommented:
Adjust your cell formatting to only display the .75
0
 
tomfarrarAuthor Commented:
It is not just formatting.  I need the number to be exactly $.75
0
 
NorieVBA ExpertCommented:
Try =ROUND(B2-B3,2) etc.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
Rob HensonFinance AnalystCommented:
Its all to do with Excel's built in level of accuracy; It will only calculate to 15 significant places.

However, that doesn't help explain why two hard-coded (as opposed to calculated) numbers that are only 3/4 significant figures (2 before decimal and 2 after decimal) subsequently create a 15th significant figure, the 2 at the very end.
0
 
ShumsDistinguished Expert - 2017Commented:
Try:
=ROUNDDOWN(B2-B3,2)

Open in new window

0
 
NorieVBA ExpertCommented:
This is a problem caused by the internal representation of floating point numbers, which uses a fixed number of binary digits to represent a decimal number.

Some decimal numbers can't be represented exactly in binary, resulting in small roundoff errors.

PS It's not just Excel.:)
1
 
Paul SauvéRetiredCommented:
Why doesn't one of these formulas in attached spreadsheet columns B,C,D,E give me an answer of $.75?  Not sure I understand why the $.750000000000002...  What am I missing?  Thanks.

select the cells and use the number format Currency―problem solved
EE_currrency.xlsx
1
 
pcelbaCommented:
The inaccuracy is caused by underlying data type which is "double" for the standard number. Double uses binary representation in 64 bits split between mantissa and exponent and the important thing is not all decimal numbers do have exact conversion to binary representation. Some (for us) simple numbers containing fractional part are converted to periodic binary number so certain rounding must happen.

Currency data type uses big integer (64 bits) with emulated 4 decimals obviously. So the accuracy is ensured if the 4 decimal places are enough. I am not sure whether Excel uses currency data type or just rounds the values.

Many programming languages and database engines (and some CPUs) use decimal data type which calculates and stores the decimal representation of the number without any conversion. Such data type cannot loose accuracy the same way as double. Of course calculations are much slower when used CPU does not support decimal data type.

Update: Sorry, the first paragraph was explained by Norie already.
0
 
tomfarrarAuthor Commented:
There were good thoughts by all, but Norie was first and most concise.  Thank you all.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now