Solved

Convert a REAL datatype to DECIMAL without losing precision

Posted on 2010-11-18
9
2,192 Views
Last Modified: 2012-05-10
Hello experts!

We have a C# applicaiton that uses double datatypes to calcualte money with Exchange rate processing. The exchange rate is currently defined in SQL Server 2008 as a REAL data type which is an 'approximate' datatype as defined by MS.

 http://msdn.microsoft.com/en-us/library/ms187912.aspx

So, I need to convert the REAL datatype to an exact datatype like DECIMAL. However, when applying the ALTER COLMN statement below I am losing precision.

Is there a DATATYPE definition that will provide the result=1.0500000000000000 ?? thanks
---------------------
CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

-- Try tyo convert field1 from a REAL to exacat precision datatype --

ALTER TABLE table1
--ALTER COLUMN field1 float(53) /* returns=1.0499999995231628 */
-- ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
--ALTER COLUMN field1 decimal(38,8) /* returns=1.04999995 */
ALTER COLUMN field1 decimal(38,7) /* returns=1.0500000 */

-- I want the value to be 16 digits to the right but be convert with exact precision
-- Can any datatype definition give me 1.0500000000000000 when converting from a REAL?
select * from table1
DROP TABLE table1
0
Comment
Question by:Mez4343
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 34169074
the problem with real is that it has accuracy problems as you have found out.
The column decimal(38,16) is correct, but the real value 1.05 with all its inaccuracy is revealed by a straight conversion to decimal.
check

select convert(decimal(30,16),convert(real,1.05))

what you actually need to do is alter to decimal, then FIX the rounding issue

update table1 set field1 = round(field1, 8)
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34169473
thanks for the input. When I use the ROUND statement after the ALTER to DECIMAL it will just truncate the remaining 8 digits. So that wont work for me. I just want a DECIMAL value with 8 digits of zeros appended to end of value.

This is with the rounding but I get 1.0499999500000000.
I need 1.0500000000000000 so maybe I have to manipulate field1 prior to converting to DECIMAL?

CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1


ALTER TABLE table1
ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
update table1 set field1 = round(field1, 8)
select * from table1
DROP TABLE table1
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34170397
So round it to 6 places. round(field1, 6)
0
Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

 
LVL 37

Accepted Solution

by:
TommySzalapski earned 167 total points
ID: 34170671
See. If the data is in a REAL to begin with then you can't "fix" it by converting it to a decimal. The precision has been lost already since it was in a REAL. You can't get the precision back. It's gone. So all you can do is round it enough to get it to look like what you want.

>>"I want the value to be 16 digits to the right but be convert with exact precision"
This is impossible. It's stored as a real so you are going to get about 7 digits of accurate precision. That's it. No way around it.

If you want all the zeros, then round it and then display it with that many digits of "accuracy".

The only way to really fix it would be to change the data types in the database and put all the values back in the way you want them. But what do you need more than 6 digits of accuracy for?
0
 
LVL 44

Assisted Solution

by:Arthur_Wood
Arthur_Wood earned 167 total points
ID: 34172173
see my answer to this question:

http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_23977342.html

It has to do with how Decimal values are actually stored in a digital computer's memory, and the in-exact representation of decimal fractions.

AW
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34173845
Tommy>>But what do you need more than 6 digits of accuracy for? <<
Our application uses double precision in the calculation code which is why I need to have 16 digit precision on this rate field. Here is an example

EUR/ USD
1/1 (converts fine)
1.05/0.952381 (stored as real)
1.0500000000000000/0.9523809523809524 (as double precision)

5,000,000 USD * .0952381 = 4,761,905 (converts USD to EUR)
4,761,905 * 1.05 = 5,000,000.25 (converts back to USD) but is off by 25 cents.
--------------------
I understand that when storing the values as REAL I automatically lose the 9 digits to the right as shown in the  USD rate above. So there is no DATATYPE defintion or built-in function that can get around this problem I guess.

Can anyone help with a T-SQL procedure that can:
1) SELECT and convert the REAL values into DECIMAL(38,16) temp field
2) ALTER field1 as DECIMAL(38,16)
3)  and then run an UPDATE statement to replace the converted values with the calcualted values in temp storage

This may be impossible, just thought I would as the experts.

Arthur, thanks for that textbook explanation. I never new that after 25 years of doing this!
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34174206
If you really need that level of precision stored in the table you could store it as a string out to as many places as you want.
0
 
LVL 58

Assisted Solution

by:cyberkiwi
cyberkiwi earned 166 total points
ID: 34176952
Mez
Real will give about 10 significant figures precision, whereas decimal 38,16 gives you far, far more, to 38 significant figures.
That is why for currency conversion, I suggested convert -> decimal, then round at the 8th digit after decimal, plus the two before is about as far as REAL is accurate.

I say significant figures because significant figures is NOT decimal places.
0
 
LVL 10

Author Closing Comment

by:Mez4343
ID: 34193083
Thanks. I decided to write a conversion program that will report on any truncated rate values (values with non-zero in all 7 digits to right of decimal) and have the customer review that rate and reinput. That will recalcualte the rate with full double precision with our new code changes.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We are taking giant steps in technological advances in the field of wireless telephony. At just 10 years since the advent of smartphones, it is crucial to examine the benefits and disadvantages that have been report to us.
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question