Solved

problem with float datatype and rounding and subnormal numbers

Posted on 2010-09-19
10
310 Views
Last Modified: 2012-06-27
Hi,

I am working with a SQL database that returns some amount. The data types for amount column is float and rounding is alredy applied in the calculation.

The problem that I have at the moment is that I need to display all the record that have their amount as:

tblX.ColAmount > 0

the problem is that the ColAmount contains values such as: 0.000123-e12

I am not sure what this exactly does but the values are not returning anymore.

the questions are:
what is exactly this value 0.000123-e12 is and why they are stored in this way in the database ?

How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?


Thanks in advance

0
Comment
Question by:shmz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
10 Comments
 
LVL 92

Expert Comment

by:Patrick Matthews
ID: 33711621
That is a very tiny, but non-zero number: 0.000123 times 10 ^ -12.Since it is greater than zero, it is getting returned by your query.In any event, you should try avoiding float if you possibly can, as it is an inherently imprecise data type.  Use decimal/numeric instead, which allows you to specify how many decimal places to carry, or use money (which uses 4 decimal places).
0
 
LVL 32

Expert Comment

by:Ephraim Wangoya
ID: 33711624
Your rounding is resulting to zero thats why you get no records

if  tblX.ColAmountis 0.000123-e12
round(tblX.ColAmount, 3) -> 0
round(tblX.ColAmount, 4) -> 1
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712704
The data types for amount column is float and rounding is already applied in the calculation.
How  should I define this comparison tblX.ColAmount > 0 to get missing records displayed?

Don't apply the rounding.
I think your number is  select 0.000123e-12  i.e. e-12, not -e12 right?

You need to round to 16 to even see any value from such a tiny number.

Round(tblX.ColAmount, 16)
0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 

Author Comment

by:shmz
ID: 33712840
data are already imported from another database with rounding already applied.
I am just displaying data in a report.
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33712889
Can you elaborate on what you mean by this?

"I am not sure what this exactly does but the values are not returning anymore."

Do you expect some values with value of 0, that are now not showing anymore?
If the numbers are already rounded, where do you see this "0.000123e-12" ?
0
 

Author Comment

by:shmz
ID: 33713543
okay:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                     0
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 

Author Comment

by:shmz
ID: 33713567
corecting my previous message:

ID                        ColAmount
--------              -----------------
234                    0.000123e-12
235                    0.001
236                    0.000124e-16
237                    0.000155e-13

I search in the query for any colAmounts > 0.0001 and as a result only record with Id 235 returns while
I like to see all above records...
0
 
LVL 58

Expert Comment

by:cyberkiwi
ID: 33713583
To give you an idea of how big (small) the numbers are

The following table should tell you why > 0.0001 only returns one match.
ID	ColAmount  Actual
234	1.23E-16   0.0000000000000001230000
235	0.001      0.0010000000000000000000
236	1.24E-20   0.0000000000000000000124
237	1.55E-17   0.0000000000000000155000

Open in new window

0
 
LVL 58

Accepted Solution

by:
cyberkiwi earned 500 total points
ID: 33713593
colAmounts > 0 should normally still return all of them.

See example here, all accounted for.

with tblX(id, colAmount) as (
select convert(int,234), convert(float,1.23E-16) union all
select 235,0.001 union all
select 236,1.24E-20 union all
select 237,1.55E-17)

select * from tblX where colAmount > 0
0
 

Author Closing Comment

by:shmz
ID: 33749909
good answer
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

So every once in a while at work I am asked to export data from one table and insert it into another on a different server.  I hate doing this.  There's so many different tables and data types.  Some column data needs quoted and some doesn't.  What …
In this article I will describe the Backup & Restore method as one possible migration process and I will add the extra tasks needed for an upgrade when and where is applied so it will cover all.
NetCrunch network monitor is a highly extensive platform for network monitoring and alert generation. In this video you'll see a live demo of NetCrunch with most notable features explained in a walk-through manner. You'll also get to know the philos…
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …

617 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question