Solved

# Convert a REAL datatype to DECIMAL without losing precision

Posted on 2010-11-18
2,028 Views
Last Modified: 2012-05-10
Hello experts!

We have a C# applicaiton that uses double datatypes to calcualte money with Exchange rate processing. The exchange rate is currently defined in SQL Server 2008 as a REAL data type which is an 'approximate' datatype as defined by MS.

http://msdn.microsoft.com/en-us/library/ms187912.aspx

So, I need to convert the REAL datatype to an exact datatype like DECIMAL. However, when applying the ALTER COLMN statement below I am losing precision.

Is there a DATATYPE definition that will provide the result=1.0500000000000000 ?? thanks
---------------------
CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

-- Try tyo convert field1 from a REAL to exacat precision datatype --

ALTER TABLE table1
--ALTER COLUMN field1 float(53) /* returns=1.0499999995231628 */
-- ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
--ALTER COLUMN field1 decimal(38,8) /* returns=1.04999995 */
ALTER COLUMN field1 decimal(38,7) /* returns=1.0500000 */

-- I want the value to be 16 digits to the right but be convert with exact precision
-- Can any datatype definition give me 1.0500000000000000 when converting from a REAL?
select * from table1
DROP TABLE table1
0
Question by:Mez4343
• 3
• 3
• 2
• +1
9 Comments

LVL 58

Expert Comment

the problem with real is that it has accuracy problems as you have found out.
The column decimal(38,16) is correct, but the real value 1.05 with all its inaccuracy is revealed by a straight conversion to decimal.
check

select convert(decimal(30,16),convert(real,1.05))

what you actually need to do is alter to decimal, then FIX the rounding issue

update table1 set field1 = round(field1, 8)
0

LVL 10

Author Comment

thanks for the input. When I use the ROUND statement after the ALTER to DECIMAL it will just truncate the remaining 8 digits. So that wont work for me. I just want a DECIMAL value with 8 digits of zeros appended to end of value.

This is with the rounding but I get 1.0499999500000000.
I need 1.0500000000000000 so maybe I have to manipulate field1 prior to converting to DECIMAL?

CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

ALTER TABLE table1
ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
update table1 set field1 = round(field1, 8)
select * from table1
DROP TABLE table1
0

LVL 37

Expert Comment

So round it to 6 places. round(field1, 6)
0

LVL 37

Accepted Solution

TommySzalapski earned 167 total points
See. If the data is in a REAL to begin with then you can't "fix" it by converting it to a decimal. The precision has been lost already since it was in a REAL. You can't get the precision back. It's gone. So all you can do is round it enough to get it to look like what you want.

>>"I want the value to be 16 digits to the right but be convert with exact precision"
This is impossible. It's stored as a real so you are going to get about 7 digits of accurate precision. That's it. No way around it.

If you want all the zeros, then round it and then display it with that many digits of "accuracy".

The only way to really fix it would be to change the data types in the database and put all the values back in the way you want them. But what do you need more than 6 digits of accuracy for?
0

LVL 44

Assisted Solution

Arthur_Wood earned 167 total points
see my answer to this question:

http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_23977342.html

It has to do with how Decimal values are actually stored in a digital computer's memory, and the in-exact representation of decimal fractions.

AW
0

LVL 10

Author Comment

Tommy>>But what do you need more than 6 digits of accuracy for? <<
Our application uses double precision in the calculation code which is why I need to have 16 digit precision on this rate field. Here is an example

EUR/ USD
1/1 (converts fine)
1.05/0.952381 (stored as real)
1.0500000000000000/0.9523809523809524 (as double precision)

5,000,000 USD * .0952381 = 4,761,905 (converts USD to EUR)
4,761,905 * 1.05 = 5,000,000.25 (converts back to USD) but is off by 25 cents.
--------------------
I understand that when storing the values as REAL I automatically lose the 9 digits to the right as shown in the  USD rate above. So there is no DATATYPE defintion or built-in function that can get around this problem I guess.

Can anyone help with a T-SQL procedure that can:
1) SELECT and convert the REAL values into DECIMAL(38,16) temp field
2) ALTER field1 as DECIMAL(38,16)
3)  and then run an UPDATE statement to replace the converted values with the calcualted values in temp storage

This may be impossible, just thought I would as the experts.

Arthur, thanks for that textbook explanation. I never new that after 25 years of doing this!
0

LVL 37

Expert Comment

If you really need that level of precision stored in the table you could store it as a string out to as many places as you want.
0

LVL 58

Assisted Solution

cyberkiwi earned 166 total points
Mez
Real will give about 10 significant figures precision, whereas decimal 38,16 gives you far, far more, to 38 significant figures.
That is why for currency conversion, I suggested convert -> decimal, then round at the 8th digit after decimal, plus the two before is about as far as REAL is accurate.

I say significant figures because significant figures is NOT decimal places.
0

LVL 10

Author Closing Comment

Thanks. I decided to write a conversion program that will report on any truncated rate values (values with non-zero in all 7 digits to right of decimal) and have the customer review that rate and reinput. That will recalcualte the rate with full double precision with our new code changes.
0

## Join & Write a Comment Already a member? Login.

This is a research brief on the potential colonization of humans on Mars.
Slowly Changing Dimension Transformation component in data task flow is very useful for us to manage and control how data changes in SSIS.
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.
Viewers will learn how the fundamental information of how to create a table.

#### 771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

#### Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!