Solved

problem with float datatype and rounding and subnormal numbers

Posted on 2010-09-19
10
294 Views
Last Modified: 2012-06-27
Hi,

I am working with a SQL database that returns some amount. The data types for amount column is float and rounding is alredy applied in the calculation.

The problem that I have at the moment is that I need to display all the record that have their amount as:

tblX.ColAmount > 0

the problem is that the ColAmount contains values such as: 0.000123-e12

I am not sure what this exactly does but the values are not returning anymore.

the questions are:
what is exactly this value 0.000123-e12 is and why they are stored in this way in the database ?

How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?


Thanks in advance

0
Comment
Question by:shmz
10 Comments
 
LVL 92

Expert Comment

by:Patrick Matthews
ID: 33711621
That is a very tiny, but non-zero number: 0.000123 times 10 ^ -12.Since it is greater than zero, it is getting returned by your query.In any event, you should try avoiding float if you possibly can, as it is an inherently imprecise data type.  Use decimal/numeric instead, which allows you to specify how many decimal places to carry, or use money (which uses 4 decimal places).
0
 
LVL 32

Expert Comment

by:ewangoya
ID: 33711624
Your rounding is resulting to zero thats why you get no records

if  tblX.ColAmountis 0.000123-e12
round(tblX.ColAmount, 3) -> 0
round(tblX.ColAmount, 4) -> 1
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712704
The data types for amount column is float and rounding is already applied in the calculation.
How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?

Don't apply the rounding.
I think your number is  select 0.000123e-12  i.e. e-12, not -e12 right?

You need to round to 16 to even see any value from such a tiny number.

Round(tblX.ColAmount, 16)
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Author Comment

by:shmz
ID: 33712840
data are already imported from another database with rounding already applied.
I am just displaying data in a report.
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712889
Can you elaborate on what you mean by this?

"I am not sure what this exactly does but the values are not returning anymore."

Do you expect some values with value of 0, that are now not showing anymore?
If the numbers are already rounded, where do you see this "0.000123e-12" ?
0
 

Author Comment

by:shmz
ID: 33713543
okay:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                     0
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 

Author Comment

by:shmz
ID: 33713567
corecting my previous message:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                    0.001
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33713583
To give you an idea of how big (small) the numbers are

The following table should tell you why > 0.0001 only returns one match.
ID	ColAmount  Actual
234	1.23E-16   0.0000000000000001230000
235	0.001      0.0010000000000000000000
236	1.24E-20   0.0000000000000000000124
237	1.55E-17   0.0000000000000000155000

Open in new window

0
 
LVL 58

Accepted Solution

by:
cyberkiwi earned 500 total points
ID: 33713593
colAmounts > 0 should normally still return all of them.

See example here, all accounted for.

with tblX(id, colAmount) as (
select convert(int,234), convert(float,1.23E-16) union all
select 235,0.001 union all
select 236,1.24E-20 union all
select 237,1.55E-17)

select * from tblX where colAmount > 0
0
 

Author Closing Comment

by:shmz
ID: 33749909
good answer
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Divide by zero error encountered. 2 43
ms sql + top 1 for each customer 3 57
Sql Query join multiple table and distinct records 7 31
Sql Server group by 10 45
Recently, when I was asked to create a new SQL 2005 cluster, Microsoft released a new service pack for MS SQL 2005 what is Service Pack 3. When I finished the installation of MS SQL 2005 I found myself troubled why the installation of SP3 failed …
When writing XML code a very difficult part is when we like to remove all the elements or attributes from the XML that have no data. I would like to share a set of recursive MSSQL stored procedures that I have made to remove those elements from …
Microsoft Active Directory, the widely used IT infrastructure, is known for its high risk of credential theft. The best way to test your Active Directory’s vulnerabilities to pass-the-ticket, pass-the-hash, privilege escalation, and malware attacks …

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question