Simple calculation in VB Script

Hi,

I am doing a simple calculation in my script and the answer is not coming back as expected for some reason.

see image:  Calculation



The calculation is simply:

1.067 - 1

Which should equal 0.67

But I get the answer:  6.6999999999999

I do not know why!!
SmashAndGrabAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Rgonzo1971Commented:
HI,

You are trying to calculate with strings

pls try somenting like this

Result = CStr(CDbl("1.65") - CDbl("1"))

Regards
0
MacroShadowCommented:
The behavior you encountered is due to a floating point error. You can correct it by converting the number to decimal.
Transaction.Screen.Item("NONMREVIEW")Fields.Item(NEWTOTQTY").Value = CDec(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value) - Transaction.Screen.Item("NONMREVIEW")Fields.Item(QTY").Value

Open in new window

0
SmashAndGrabAuthor Commented:
Hi,

I have tried both the methods and neither worked unfortunalty :(

Method 1

Result = CStr(CDbl("1.65") - CDbl("1"))      This returned the 6.999999

Method 2

Returned this error:

error.JPG
0
Exploring SharePoint 2016

Explore SharePoint 2016, the web-based, collaborative platform that integrates with Microsoft Office to provide intranets, secure document management, and collaboration so you can develop your online and offline capabilities.

Rgonzo1971Commented:
Hi,

pls try

Result = FormatNumber(CDbl("1.067") - CDbl("1"), 3)
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MacroShadowCommented:
Turns out CDec isn't supported in Asp.

Try this monstrous code, it may work (I've never used ASP.
Int(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value) + (Right(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value,Len(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value)-InStrRev(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value,".")+1)) - Transaction.Screen.Item("NONMREVIEW")Fields.Item(QTY").Value

Open in new window

0
SmashAndGrabAuthor Commented:
perfect!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VB Script

From novice to tech pro — start learning today.