Comparing decimal values in classic asp is not producing correct result

I have a pretty straightforward comparison of two values that is not giving the intended results.

The page calculates the total qty of a sale of a part from history and compares that number to what the system thinks the total qty sold is.

Basically, I have an On Hand quantity of a part.  I calculate the entire part's sales and purchase orders to get with what the system should be at, based on history, and compare the two.

Recently, we decided to change the qty field from INT to FLOAT (sql 2014 back end) to accommodate for some manufacturing measurements.

So, as of now, I have a page that has this comparison:

if cDbl(rs("CalculatedShelfOH")) <> cDbl(rs("CardOH")) then
do stuff
end if

The CalculatdShelfOH is 2.126 and the CardOH is 2.126.  But, the code is "doing the stuff" even though they match.

What am I missing / doing wrong?
dzirkelbAsked:
Who is Participating?
 
Big MontySenior Web Developer / CEO of ExchangeTree.org Commented:
i agree, it's probably a rounding issue. If you don't always know how many decimal places are going to be present, you could always convert the values to a string and do the compare:

if cStr(rs("CalculatedShelfOH")) <> cStr(rs("CardOH")) then
0
 
NorieVBA ExpertCommented:
It could be a problem with floating point precision/error/arithmetic, try rounding.

If Roundl(rs("CalculatedShelfOH"),3) <> Round(rs("CardOH"),3) Then

Open in new window

0
 
Paul MacDonaldDirector, Information SystemsCommented:
From Microsoft Documentation:
"When you work with floating-point numbers, remember that they do not always have a precise representation in memory. This could lead to unexpected results from certain operations, such as value comparison and the Mod operator. For more information, see Troubleshooting Data Types."
0
 
dzirkelbAuthor Commented:
Thanks for the suggestions.  I will go with the cStr function as I will have many pages to update, and I just don't know how many decimals will be present on each function.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.