• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 919
  • Last Modified:

Subtracting 2 Numbers Problem

I am trying to subtract 2 numbers; the numbers come from different sources (one from the Internet, the other from a table in a database where its type is decimal.

If they both have the same value, when I try to subtract them, I get a result like 7.105427357601e-015. I tried casting to int (.to_i) and it works fine but I lose the decimal part.  If I cast them to float I get a similar result to that given above.  I cannoy think of any other cast to make.

Another thing if the two numbers are 84.21 and 80.21, the difference is 3.9999999999999

Subtracting 84.21 from 80.18 produces the correct value.

Sometimes I despair.

Can anyone help?

0
witgrefe
Asked:
witgrefe
  • 4
1 Solution
 
Geert BormansCommented:
7.105427357601e-015 is a very, very, very small number,
so this is a precision issue

I assume that you don't want to loose the decimal part,
so you must have some precision in mind.

What you could do is multiply by 100, then round and the divide by 100 again.
Then you will have a precison up to 0.01

3.9999999999999 will become 4.00
and 7.105427357601e-015  will become 0

If you need more precision, multiply by thousand, round and divide again
0
 
Geert BormansCommented:
Try this

result = 3.99999999999
puts (result * 100).round.div(100).to_s

and

result = 7.105427357601e-015
puts (result * 100).round.div(100).to_s
0
 
Geert BormansCommented:
I ran into two traps here
div gives an integer result... that is not what you want
/ would return a truncated integer since the first argument is an integer
so, I need explicit casting to float first

puts ((result * 100).round.to_f / 100).to_s
0
 
witgrefeAuthor Commented:
But I need 2 decimal places, what you have suggested gives me no decimal places.  It is fine if for the 2 examples I gave, but if I subtract 80.55 from 85.7, I get  5 without any decimal place
0
 
Geert BormansCommented:
Yep, I spotted the error
please look at my last comment
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now