Solved

# problem with float datatype and rounding and subnormal numbers

Posted on 2010-09-19
304 Views
Hi,

I am working with a SQL database that returns some amount. The data types for amount column is float and rounding is alredy applied in the calculation.

The problem that I have at the moment is that I need to display all the record that have their amount as:

tblX.ColAmount > 0

the problem is that the ColAmount contains values such as: 0.000123-e12

I am not sure what this exactly does but the values are not returning anymore.

the questions are:
what is exactly this value 0.000123-e12 is and why they are stored in this way in the database ?

How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?

0
Question by:shmz
[X]
###### Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

• Help others & share knowledge
• Earn cash & points

LVL 92

Expert Comment

ID: 33711621
That is a very tiny, but non-zero number: 0.000123 times 10 ^ -12.Since it is greater than zero, it is getting returned by your query.In any event, you should try avoiding float if you possibly can, as it is an inherently imprecise data type.  Use decimal/numeric instead, which allows you to specify how many decimal places to carry, or use money (which uses 4 decimal places).
0

LVL 32

Expert Comment

ID: 33711624
Your rounding is resulting to zero thats why you get no records

if  tblX.ColAmountis 0.000123-e12
round(tblX.ColAmount, 3) -> 0
round(tblX.ColAmount, 4) -> 1
0

LVL 58

Expert Comment

ID: 33712704
The data types for amount column is float and rounding is already applied in the calculation.
How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?

Don't apply the rounding.
I think your number is  select 0.000123e-12  i.e. e-12, not -e12 right?

You need to round to 16 to even see any value from such a tiny number.

Round(tblX.ColAmount, 16)
0

Author Comment

ID: 33712840
I am just displaying data in a report.
0

LVL 58

Expert Comment

ID: 33712889
Can you elaborate on what you mean by this?

"I am not sure what this exactly does but the values are not returning anymore."

Do you expect some values with value of 0, that are now not showing anymore?
If the numbers are already rounded, where do you see this "0.000123e-12" ?
0

Author Comment

ID: 33713543
okay:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                     0
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0

Author Comment

ID: 33713567
corecting my previous message:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                    0.001
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0

LVL 58

Expert Comment

ID: 33713583
To give you an idea of how big (small) the numbers are

The following table should tell you why > 0.0001 only returns one match.
``````ID	ColAmount  Actual
234	1.23E-16   0.0000000000000001230000
235	0.001      0.0010000000000000000000
236	1.24E-20   0.0000000000000000000124
237	1.55E-17   0.0000000000000000155000
``````
0

LVL 58

Accepted Solution

cyberkiwi earned 500 total points
ID: 33713593
colAmounts > 0 should normally still return all of them.

See example here, all accounted for.

with tblX(id, colAmount) as (
select convert(int,234), convert(float,1.23E-16) union all
select 235,0.001 union all
select 236,1.24E-20 union all
select 237,1.55E-17)

select * from tblX where colAmount > 0
0

Author Closing Comment

ID: 33749909
0

## Featured Post

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

by Mark Wills Attending one of Rob Farley's seminars the other day, I heard the phrase "The Accidental DBA" and fell in love with it. It got me thinking about the plight of the newcomer to SQL Server...  So if you are the accidental DBA, or, simp…
Introduction: When running hybrid database environments, you often need to query some data from a remote db of any type, while being connected to your MS SQL Server database. Problems start when you try to combine that with some "user input" pass…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
###### Suggested Courses
Course of the Month2 days, 4 hours left to enroll