SQL Server precision differing from excel precision

Posted on 2005-04-27
Last Modified: 2012-06-21
I have developed a treasury system for a company that was using Microsoft Excel for their processing. I have moved most of the processing into SQL Server procedures and functions. The interesting thing is that for large values, the calculations in SQL Server and those in excel yield different values. Is this a data type issue? I am using DECIMAL(38,2) for all currency values. I am puzzled as to why we are getting different values after using the same formulae in excel and SQL Server.
Question by:Goodangel Matope
    LVL 20

    Expert Comment

    The decimal format stores numbers in  SS2000 as BCD (binary coded decimal) with a maximum precision of 38 digits.  Excel is using floating-point which has less precision.  However, if you perform math operations with the numbers you may find that your application introduces loss of precision (due to intermediate rounding) with either data.
    LVL 9

    Accepted Solution

    I can't say for sure without seeing the actual functions and data, but I'd make a very confident guess on it being down to rounding errors and / or data type conversions...

    take (£100.00 / 6) * 3. Logic would suggest that this would give the answer £50.00
    However, if the original value 100 is stored in a data type int, the calculation COULD go as follows...
    £100.00 / 6 = £16.66
    £16.66 * 3 =  £49.98

    It's a poor example, but I hope this illustrates how two systems could easily get two different answers to the same calculation.



    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    Database tuning – How to start and what to tune. This question is frequently asked by many people, both online and offline. There is no hard and fast rule-of-thumb for performance tuning, however, before beginning the tuning process one should a…
    Using SQL Scripts we can save all the SQL queries as files that we use very frequently on our database later point of time. This is one of the feature present under SQL Workshop in Oracle Application Express.
    Video by: Steve
    Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
    Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now