?
Solved

Convert a REAL datatype to DECIMAL without losing precision

Posted on 2010-11-18
9
Medium Priority
?
2,727 Views
Last Modified: 2012-05-10
Hello experts!

We have a C# applicaiton that uses double datatypes to calcualte money with Exchange rate processing. The exchange rate is currently defined in SQL Server 2008 as a REAL data type which is an 'approximate' datatype as defined by MS.

 http://msdn.microsoft.com/en-us/library/ms187912.aspx

So, I need to convert the REAL datatype to an exact datatype like DECIMAL. However, when applying the ALTER COLMN statement below I am losing precision.

Is there a DATATYPE definition that will provide the result=1.0500000000000000 ?? thanks
---------------------
CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

-- Try tyo convert field1 from a REAL to exacat precision datatype --

ALTER TABLE table1
--ALTER COLUMN field1 float(53) /* returns=1.0499999995231628 */
-- ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
--ALTER COLUMN field1 decimal(38,8) /* returns=1.04999995 */
ALTER COLUMN field1 decimal(38,7) /* returns=1.0500000 */

-- I want the value to be 16 digits to the right but be convert with exact precision
-- Can any datatype definition give me 1.0500000000000000 when converting from a REAL?
select * from table1
DROP TABLE table1
0
Comment
Question by:Mez4343
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 34169074
the problem with real is that it has accuracy problems as you have found out.
The column decimal(38,16) is correct, but the real value 1.05 with all its inaccuracy is revealed by a straight conversion to decimal.
check

select convert(decimal(30,16),convert(real,1.05))

what you actually need to do is alter to decimal, then FIX the rounding issue

update table1 set field1 = round(field1, 8)
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34169473
thanks for the input. When I use the ROUND statement after the ALTER to DECIMAL it will just truncate the remaining 8 digits. So that wont work for me. I just want a DECIMAL value with 8 digits of zeros appended to end of value.

This is with the rounding but I get 1.0499999500000000.
I need 1.0500000000000000 so maybe I have to manipulate field1 prior to converting to DECIMAL?

CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1


ALTER TABLE table1
ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
update table1 set field1 = round(field1, 8)
select * from table1
DROP TABLE table1
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34170397
So round it to 6 places. round(field1, 6)
0
Get MySQL database support online, now!

At Percona’s web store you can order your MySQL database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card.

 
LVL 37

Accepted Solution

by:
TommySzalapski earned 501 total points
ID: 34170671
See. If the data is in a REAL to begin with then you can't "fix" it by converting it to a decimal. The precision has been lost already since it was in a REAL. You can't get the precision back. It's gone. So all you can do is round it enough to get it to look like what you want.

>>"I want the value to be 16 digits to the right but be convert with exact precision"
This is impossible. It's stored as a real so you are going to get about 7 digits of accurate precision. That's it. No way around it.

If you want all the zeros, then round it and then display it with that many digits of "accuracy".

The only way to really fix it would be to change the data types in the database and put all the values back in the way you want them. But what do you need more than 6 digits of accuracy for?
0
 
LVL 44

Assisted Solution

by:Arthur_Wood
Arthur_Wood earned 501 total points
ID: 34172173
see my answer to this question:

http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_23977342.html

It has to do with how Decimal values are actually stored in a digital computer's memory, and the in-exact representation of decimal fractions.

AW
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34173845
Tommy>>But what do you need more than 6 digits of accuracy for? <<
Our application uses double precision in the calculation code which is why I need to have 16 digit precision on this rate field. Here is an example

EUR/ USD
1/1 (converts fine)
1.05/0.952381 (stored as real)
1.0500000000000000/0.9523809523809524 (as double precision)

5,000,000 USD * .0952381 = 4,761,905 (converts USD to EUR)
4,761,905 * 1.05 = 5,000,000.25 (converts back to USD) but is off by 25 cents.
--------------------
I understand that when storing the values as REAL I automatically lose the 9 digits to the right as shown in the  USD rate above. So there is no DATATYPE defintion or built-in function that can get around this problem I guess.

Can anyone help with a T-SQL procedure that can:
1) SELECT and convert the REAL values into DECIMAL(38,16) temp field
2) ALTER field1 as DECIMAL(38,16)
3)  and then run an UPDATE statement to replace the converted values with the calcualted values in temp storage

This may be impossible, just thought I would as the experts.

Arthur, thanks for that textbook explanation. I never new that after 25 years of doing this!
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34174206
If you really need that level of precision stored in the table you could store it as a string out to as many places as you want.
0
 
LVL 58

Assisted Solution

by:cyberkiwi
cyberkiwi earned 498 total points
ID: 34176952
Mez
Real will give about 10 significant figures precision, whereas decimal 38,16 gives you far, far more, to 38 significant figures.
That is why for currency conversion, I suggested convert -> decimal, then round at the 8th digit after decimal, plus the two before is about as far as REAL is accurate.

I say significant figures because significant figures is NOT decimal places.
0
 
LVL 10

Author Closing Comment

by:Mez4343
ID: 34193083
Thanks. I decided to write a conversion program that will report on any truncated rate values (values with non-zero in all 7 digits to right of decimal) and have the customer review that rate and reinput. That will recalcualte the rate with full double precision with our new code changes.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question