[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 392
  • Last Modified:

SQL Server precision differing from excel precision

I have developed a treasury system for a company that was using Microsoft Excel for their processing. I have moved most of the processing into SQL Server procedures and functions. The interesting thing is that for large values, the calculations in SQL Server and those in excel yield different values. Is this a data type issue? I am using DECIMAL(38,2) for all currency values. I am puzzled as to why we are getting different values after using the same formulae in excel and SQL Server.
Goodangel Matope
Goodangel Matope
1 Solution
The decimal format stores numbers in  SS2000 as BCD (binary coded decimal) with a maximum precision of 38 digits.  Excel is using floating-point which has less precision.  However, if you perform math operations with the numbers you may find that your application introduces loss of precision (due to intermediate rounding) with either data.
I can't say for sure without seeing the actual functions and data, but I'd make a very confident guess on it being down to rounding errors and / or data type conversions...

take (£100.00 / 6) * 3. Logic would suggest that this would give the answer £50.00
However, if the original value 100 is stored in a data type int, the calculation COULD go as follows...
£100.00 / 6 = £16.66
£16.66 * 3 =  £49.98

It's a poor example, but I hope this illustrates how two systems could easily get two different answers to the same calculation.



Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now