Solved

Convert a REAL datatype to DECIMAL without losing precision

Posted on 2010-11-18
9
2,099 Views
Last Modified: 2012-05-10
Hello experts!

We have a C# applicaiton that uses double datatypes to calcualte money with Exchange rate processing. The exchange rate is currently defined in SQL Server 2008 as a REAL data type which is an 'approximate' datatype as defined by MS.

 http://msdn.microsoft.com/en-us/library/ms187912.aspx

So, I need to convert the REAL datatype to an exact datatype like DECIMAL. However, when applying the ALTER COLMN statement below I am losing precision.

Is there a DATATYPE definition that will provide the result=1.0500000000000000 ?? thanks
---------------------
CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

-- Try tyo convert field1 from a REAL to exacat precision datatype --

ALTER TABLE table1
--ALTER COLUMN field1 float(53) /* returns=1.0499999995231628 */
-- ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
--ALTER COLUMN field1 decimal(38,8) /* returns=1.04999995 */
ALTER COLUMN field1 decimal(38,7) /* returns=1.0500000 */

-- I want the value to be 16 digits to the right but be convert with exact precision
-- Can any datatype definition give me 1.0500000000000000 when converting from a REAL?
select * from table1
DROP TABLE table1
0
Comment
Question by:Mez4343
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 34169074
the problem with real is that it has accuracy problems as you have found out.
The column decimal(38,16) is correct, but the real value 1.05 with all its inaccuracy is revealed by a straight conversion to decimal.
check

select convert(decimal(30,16),convert(real,1.05))

what you actually need to do is alter to decimal, then FIX the rounding issue

update table1 set field1 = round(field1, 8)
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34169473
thanks for the input. When I use the ROUND statement after the ALTER to DECIMAL it will just truncate the remaining 8 digits. So that wont work for me. I just want a DECIMAL value with 8 digits of zeros appended to end of value.

This is with the rounding but I get 1.0499999500000000.
I need 1.0500000000000000 so maybe I have to manipulate field1 prior to converting to DECIMAL?

CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1


ALTER TABLE table1
ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
update table1 set field1 = round(field1, 8)
select * from table1
DROP TABLE table1
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34170397
So round it to 6 places. round(field1, 6)
0
 
LVL 37

Accepted Solution

by:
TommySzalapski earned 167 total points
ID: 34170671
See. If the data is in a REAL to begin with then you can't "fix" it by converting it to a decimal. The precision has been lost already since it was in a REAL. You can't get the precision back. It's gone. So all you can do is round it enough to get it to look like what you want.

>>"I want the value to be 16 digits to the right but be convert with exact precision"
This is impossible. It's stored as a real so you are going to get about 7 digits of accurate precision. That's it. No way around it.

If you want all the zeros, then round it and then display it with that many digits of "accuracy".

The only way to really fix it would be to change the data types in the database and put all the values back in the way you want them. But what do you need more than 6 digits of accuracy for?
0
Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

 
LVL 44

Assisted Solution

by:Arthur_Wood
Arthur_Wood earned 167 total points
ID: 34172173
see my answer to this question:

http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_23977342.html

It has to do with how Decimal values are actually stored in a digital computer's memory, and the in-exact representation of decimal fractions.

AW
0
 
LVL 10

Author Comment

by:Mez4343
ID: 34173845
Tommy>>But what do you need more than 6 digits of accuracy for? <<
Our application uses double precision in the calculation code which is why I need to have 16 digit precision on this rate field. Here is an example

EUR/ USD
1/1 (converts fine)
1.05/0.952381 (stored as real)
1.0500000000000000/0.9523809523809524 (as double precision)

5,000,000 USD * .0952381 = 4,761,905 (converts USD to EUR)
4,761,905 * 1.05 = 5,000,000.25 (converts back to USD) but is off by 25 cents.
--------------------
I understand that when storing the values as REAL I automatically lose the 9 digits to the right as shown in the  USD rate above. So there is no DATATYPE defintion or built-in function that can get around this problem I guess.

Can anyone help with a T-SQL procedure that can:
1) SELECT and convert the REAL values into DECIMAL(38,16) temp field
2) ALTER field1 as DECIMAL(38,16)
3)  and then run an UPDATE statement to replace the converted values with the calcualted values in temp storage

This may be impossible, just thought I would as the experts.

Arthur, thanks for that textbook explanation. I never new that after 25 years of doing this!
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 34174206
If you really need that level of precision stored in the table you could store it as a string out to as many places as you want.
0
 
LVL 58

Assisted Solution

by:cyberkiwi
cyberkiwi earned 166 total points
ID: 34176952
Mez
Real will give about 10 significant figures precision, whereas decimal 38,16 gives you far, far more, to 38 significant figures.
That is why for currency conversion, I suggested convert -> decimal, then round at the 8th digit after decimal, plus the two before is about as far as REAL is accurate.

I say significant figures because significant figures is NOT decimal places.
0
 
LVL 10

Author Closing Comment

by:Mez4343
ID: 34193083
Thanks. I decided to write a conversion program that will report on any truncated rate values (values with non-zero in all 7 digits to right of decimal) and have the customer review that rate and reinput. That will recalcualte the rate with full double precision with our new code changes.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever thought of installing a power system that generates solar electricity to power your house? Some may say yes, while others may tell me no. But have you noticed that people around you are now considering installing such systems in their …
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.

932 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now