LuckyLucks
asked on
Precision n Scale in Real and Double
Hi EEE,
I am hoping someone can explain to me the real and Double/Float datatypes in terms of precision and scale. Precision is defined as no of total digits in the number and scale as the total number of digits on the right of decimal point.
Does Real (defined as a single precision floating point number in DB2 docs) only have a single digit on the left hand side of decimal?
Does Double/Float (defined as a double precision floating point number) have only 2 digits to the left of decimal?
thx
I am hoping someone can explain to me the real and Double/Float datatypes in terms of precision and scale. Precision is defined as no of total digits in the number and scale as the total number of digits on the right of decimal point.
Does Real (defined as a single precision floating point number in DB2 docs) only have a single digit on the left hand side of decimal?
Does Double/Float (defined as a double precision floating point number) have only 2 digits to the left of decimal?
thx
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Hi Lucky,
You're close.
float and double use the native hardware and can not be size delimited. A float is a single-precision binary value and a double is a double-precision binary value.
If you want to define the size, you'll have to use DECIMAL. Keep in mind that a DECIMAL (5,3) indicates that the value is 5 total digits in length, the 3 rightmost are to the right of the decimal point. DB2 doesn't store the decimal point, just the digit so it is the column definition that indicates where the decimal point goes.
For a DECIMAL (5,3), an internal value of 12345 is a user value of 12.345. 00001 is 00.001. You can't shift the decimal point when it's convenient.
Kent
You're close.
float and double use the native hardware and can not be size delimited. A float is a single-precision binary value and a double is a double-precision binary value.
If you want to define the size, you'll have to use DECIMAL. Keep in mind that a DECIMAL (5,3) indicates that the value is 5 total digits in length, the 3 rightmost are to the right of the decimal point. DB2 doesn't store the decimal point, just the digit so it is the column definition that indicates where the decimal point goes.
For a DECIMAL (5,3), an internal value of 12345 is a user value of 12.345. 00001 is 00.001. You can't shift the decimal point when it's convenient.
Kent
ASKER
I have posed some examples below, kindly check those interpretations.
real(5) - see float(5) below
real(5,2) - see float(5,2) below
float(5) - We can assign valid values as 123.45 as well as 12.345. Invalid values are 12.3456 and 123.456, correct?
float(5,2) - We can assign valid values as 123.45. Invalid values are 123.456, correct?
double(5) - We can assign valid values as 12345.0 or 123.45 or 12.345? (total of 5 digits, no bar on the number of digits to right of decimal as long as total digits dont exceed 5?)
double(5,3) - We can assign valid values as 12.345 or 123.45, correct? (total of 5 digits and a max of 3 to right of decimal)