[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3093
  • Last Modified:

Convert a REAL datatype to DECIMAL without losing precision

Hello experts!

We have a C# applicaiton that uses double datatypes to calcualte money with Exchange rate processing. The exchange rate is currently defined in SQL Server 2008 as a REAL data type which is an 'approximate' datatype as defined by MS.

 http://msdn.microsoft.com/en-us/library/ms187912.aspx

So, I need to convert the REAL datatype to an exact datatype like DECIMAL. However, when applying the ALTER COLMN statement below I am losing precision.

Is there a DATATYPE definition that will provide the result=1.0500000000000000 ?? thanks
---------------------
CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1

-- Try tyo convert field1 from a REAL to exacat precision datatype --

ALTER TABLE table1
--ALTER COLUMN field1 float(53) /* returns=1.0499999995231628 */
-- ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
--ALTER COLUMN field1 decimal(38,8) /* returns=1.04999995 */
ALTER COLUMN field1 decimal(38,7) /* returns=1.0500000 */

-- I want the value to be 16 digits to the right but be convert with exact precision
-- Can any datatype definition give me 1.0500000000000000 when converting from a REAL?
select * from table1
DROP TABLE table1
0
Mez4343
Asked:
Mez4343
  • 3
  • 3
  • 2
  • +1
3 Solutions
 
cyberkiwiCommented:
the problem with real is that it has accuracy problems as you have found out.
The column decimal(38,16) is correct, but the real value 1.05 with all its inaccuracy is revealed by a straight conversion to decimal.
check

select convert(decimal(30,16),convert(real,1.05))

what you actually need to do is alter to decimal, then FIX the rounding issue

update table1 set field1 = round(field1, 8)
0
 
Mez4343Author Commented:
thanks for the input. When I use the ROUND statement after the ALTER to DECIMAL it will just truncate the remaining 8 digits. So that wont work for me. I just want a DECIMAL value with 8 digits of zeros appended to end of value.

This is with the rounding but I get 1.0499999500000000.
I need 1.0500000000000000 so maybe I have to manipulate field1 prior to converting to DECIMAL?

CREATE TABLE table1 (field1 real);
INSERT table1 select ('1.05')

SELECT * from table1


ALTER TABLE table1
ALTER COLUMN field1 decimal(38,16) /* returns=1.0499999995231628 */
update table1 set field1 = round(field1, 8)
select * from table1
DROP TABLE table1
0
 
TommySzalapskiCommented:
So round it to 6 places. round(field1, 6)
0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 
TommySzalapskiCommented:
See. If the data is in a REAL to begin with then you can't "fix" it by converting it to a decimal. The precision has been lost already since it was in a REAL. You can't get the precision back. It's gone. So all you can do is round it enough to get it to look like what you want.

>>"I want the value to be 16 digits to the right but be convert with exact precision"
This is impossible. It's stored as a real so you are going to get about 7 digits of accurate precision. That's it. No way around it.

If you want all the zeros, then round it and then display it with that many digits of "accuracy".

The only way to really fix it would be to change the data types in the database and put all the values back in the way you want them. But what do you need more than 6 digits of accuracy for?
0
 
Arthur_WoodCommented:
see my answer to this question:

http://www.experts-exchange.com/Programming/Languages/C_Sharp/Q_23977342.html

It has to do with how Decimal values are actually stored in a digital computer's memory, and the in-exact representation of decimal fractions.

AW
0
 
Mez4343Author Commented:
Tommy>>But what do you need more than 6 digits of accuracy for? <<
Our application uses double precision in the calculation code which is why I need to have 16 digit precision on this rate field. Here is an example

EUR/ USD
1/1 (converts fine)
1.05/0.952381 (stored as real)
1.0500000000000000/0.9523809523809524 (as double precision)

5,000,000 USD * .0952381 = 4,761,905 (converts USD to EUR)
4,761,905 * 1.05 = 5,000,000.25 (converts back to USD) but is off by 25 cents.
--------------------
I understand that when storing the values as REAL I automatically lose the 9 digits to the right as shown in the  USD rate above. So there is no DATATYPE defintion or built-in function that can get around this problem I guess.

Can anyone help with a T-SQL procedure that can:
1) SELECT and convert the REAL values into DECIMAL(38,16) temp field
2) ALTER field1 as DECIMAL(38,16)
3)  and then run an UPDATE statement to replace the converted values with the calcualted values in temp storage

This may be impossible, just thought I would as the experts.

Arthur, thanks for that textbook explanation. I never new that after 25 years of doing this!
0
 
TommySzalapskiCommented:
If you really need that level of precision stored in the table you could store it as a string out to as many places as you want.
0
 
cyberkiwiCommented:
Mez
Real will give about 10 significant figures precision, whereas decimal 38,16 gives you far, far more, to 38 significant figures.
That is why for currency conversion, I suggested convert -> decimal, then round at the 8th digit after decimal, plus the two before is about as far as REAL is accurate.

I say significant figures because significant figures is NOT decimal places.
0
 
Mez4343Author Commented:
Thanks. I decided to write a conversion program that will report on any truncated rate values (values with non-zero in all 7 digits to right of decimal) and have the customer review that rate and reinput. That will recalcualte the rate with full double precision with our new code changes.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
  • 3
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now