Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

problem with float datatype and rounding and subnormal numbers

Posted on 2010-09-19
10
Medium Priority
?
315 Views
Last Modified: 2012-06-27
Hi,

I am working with a SQL database that returns some amount. The data types for amount column is float and rounding is alredy applied in the calculation.

The problem that I have at the moment is that I need to display all the record that have their amount as:

tblX.ColAmount > 0

the problem is that the ColAmount contains values such as: 0.000123-e12

I am not sure what this exactly does but the values are not returning anymore.

the questions are:
what is exactly this value 0.000123-e12 is and why they are stored in this way in the database ?

How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?


Thanks in advance

0
Comment
Question by:shmz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
10 Comments
 
LVL 93

Expert Comment

by:Patrick Matthews
ID: 33711621
That is a very tiny, but non-zero number: 0.000123 times 10 ^ -12.Since it is greater than zero, it is getting returned by your query.In any event, you should try avoiding float if you possibly can, as it is an inherently imprecise data type.  Use decimal/numeric instead, which allows you to specify how many decimal places to carry, or use money (which uses 4 decimal places).
0
 
LVL 32

Expert Comment

by:Ephraim Wangoya
ID: 33711624
Your rounding is resulting to zero thats why you get no records

if  tblX.ColAmountis 0.000123-e12
round(tblX.ColAmount, 3) -> 0
round(tblX.ColAmount, 4) -> 1
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712704
The data types for amount column is float and rounding is already applied in the calculation.
How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?

Don't apply the rounding.
I think your number is  select 0.000123e-12  i.e. e-12, not -e12 right?

You need to round to 16 to even see any value from such a tiny number.

Round(tblX.ColAmount, 16)
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 

Author Comment

by:shmz
ID: 33712840
data are already imported from another database with rounding already applied.
I am just displaying data in a report.
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712889
Can you elaborate on what you mean by this?

"I am not sure what this exactly does but the values are not returning anymore."

Do you expect some values with value of 0, that are now not showing anymore?
If the numbers are already rounded, where do you see this "0.000123e-12" ?
0
 

Author Comment

by:shmz
ID: 33713543
okay:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                     0
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 

Author Comment

by:shmz
ID: 33713567
corecting my previous message:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                    0.001
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33713583
To give you an idea of how big (small) the numbers are

The following table should tell you why > 0.0001 only returns one match.
ID	ColAmount  Actual
234	1.23E-16   0.0000000000000001230000
235	0.001      0.0010000000000000000000
236	1.24E-20   0.0000000000000000000124
237	1.55E-17   0.0000000000000000155000

Open in new window

0
 
LVL 58

Accepted Solution

by:
cyberkiwi earned 2000 total points
ID: 33713593
colAmounts > 0 should normally still return all of them.

See example here, all accounted for.

with tblX(id, colAmount) as (
select convert(int,234), convert(float,1.23E-16) union all
select 235,0.001 union all
select 236,1.24E-20 union all
select 237,1.55E-17)

select * from tblX where colAmount > 0
0
 

Author Closing Comment

by:shmz
ID: 33749909
good answer
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

by Mark Wills Attending one of Rob Farley's seminars the other day, I heard the phrase "The Accidental DBA" and fell in love with it. It got me thinking about the plight of the newcomer to SQL Server...  So if you are the accidental DBA, or, simp…
Use this article to create a batch file to backup a Microsoft SQL Server database to a Windows folder.  The folder can be on the local hard drive or on a network share.  This batch file will query the SQL server to get the current date & time and wi…
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question