Solved

SQL Server decimal calculations showing odd results

Posted on 2012-03-22
5
464 Views
Last Modified: 2012-03-26
Consider the script below ran in Management Studio on SQLServer 2008:

          declare @N1 decimal(38,12)
          declare @N2 decimal(38,12)
          begin        
            set @N1=1111223333444455556666777.111222233
            set @N2=1111223333444455556666777.111222233
            select @N1/@N2  --expected result
            select @N1*cast(-1 as decimal(38,12))/@N2
            select @N1*(-1*(1.0))/@N2      
            select @N1*(-1.0/@N2)    
            select 1.0/@N2
          end;

The results are:
1.000000
-0.999999
-0.999999
-0.888979
0.0000000000000000000000008

Can anyone explain these results? Our problem is our software has to parse user friendly representation of field and number summing into sql friendly script which in turn gives correct and consistent results. We do need to deal with large number and many decimal places, hence using decimal(38,12). We are aware of the need to make the nominator of a division into a decimal number otherwise the result is assumed to be an integer,
ie
select 2/5 results in 0, and select 2.0/5 results in 0.4

We have also noticed that SQLServer doesn't always apply BODMAS calculation ordering. So, is there a set of rules that can be followed to achieve reliable results?

Many thanks
Alex
0
Comment
Question by:stbdevelopment
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
5 Comments
 
LVL 18

Assisted Solution

by:deighton
deighton earned 50 total points
ID: 37753166
I think you are overflowing the maximum size of a float
0
 
LVL 42

Assisted Solution

by:dqmq
dqmq earned 300 total points
ID: 37753285
Interesting...seems like the more precision you give, the less you get back.  

         declare @N1 decimal(38,12)
            set @N1=1.111122223333
            select @N1
            select @N1/1
            select @N1/1.0
          select @N1/1.00
          select @N1/1.000
            select @N1/1.00000000000000000
            select @N1/cast(1 as decimal(38,12))
            select @N1/cast(1 as decimal(38,3))

-------------------

1.111122223333

1.111122223333

1.11112222333

1.1111222233

1.111122223

1.111122

1.111122

1.111122223
0
 

Author Comment

by:stbdevelopment
ID: 37753495
Thank you dqmq. That really highlights the problem we are having. However what is the solution?
Do we need to restrict the numbers used in calculations to so many decimal places and then cast them to a larger number of decimal places? But then the results would be the same. I can't see a way out of this.
0
 
LVL 42

Accepted Solution

by:
dqmq earned 300 total points
ID: 37754117
I don't know.  Still trying to make sense of it:

For more weirdness, try this:


           declare @N1 decimal(38,12)
            set @N1=   1.111122223333
            select @N1
          select @N1/1.0
          select @N1/1.000000000000
            select 1.111122223333/1.000000000000
            select 1.111122223333/1.0
0
 
LVL 2

Assisted Solution

by:winston33
winston33 earned 50 total points
ID: 37758673
I'm sorry if this does not completely answer your question, but I generally complete the equation using the native datatypes (assuming the two numbers are using the same datatype) and then convert the result to the desired data type. The only statement below where this does not apply is the third because the "1" needs to be converted prior to the division. I cannot explain WHY this works, but I generally get the proper results. For example, using your example:

          declare @N1 decimal(38,12)
          declare @N2 decimal(38,12)
          begin        
            set @N1=1111223333444455556666777.111222233
            set @N2=1111223333444455556666777.111222233
            select CONVERT(DECIMAL(38,12), @N1/@N2)  --expected result            
            select CONVERT(DECIMAL(38,12), (@N1*-1)/@N2)            
            select CONVERT(DECIMAL(38,12), CONVERT(DECIMAL(38,12), 1)/@N2)
          end;


1.000000000000

-1.000000000000

0.000000000000 --> This is actually the correct result, I think, given the precision and scale.

Anyway, hope this helps a little.
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
SQL query for highest sequence 4 63
Help needed in sql query 4 29
Sql case statement to calculate totals 5 35
Trouble installing msi file with msiexe.exe 2 18
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Using examples as well as descriptions, and references to Books Online, show the documentation available for datatypes, explain the available data types and show how data can be passed into and out of variables.

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question