• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 64
  • Last Modified:

Simple calculation in VB Script

Hi,

I am doing a simple calculation in my script and the answer is not coming back as expected for some reason.

see image:  Calculation



The calculation is simply:

1.067 - 1

Which should equal 0.67

But I get the answer:  6.6999999999999

I do not know why!!
0
SmashAndGrab
Asked:
SmashAndGrab
  • 2
  • 2
  • 2
1 Solution
 
Rgonzo1971Commented:
HI,

You are trying to calculate with strings

pls try somenting like this

Result = CStr(CDbl("1.65") - CDbl("1"))

Regards
0
 
MacroShadowCommented:
The behavior you encountered is due to a floating point error. You can correct it by converting the number to decimal.
Transaction.Screen.Item("NONMREVIEW")Fields.Item(NEWTOTQTY").Value = CDec(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value) - Transaction.Screen.Item("NONMREVIEW")Fields.Item(QTY").Value

Open in new window

0
 
SmashAndGrabAuthor Commented:
Hi,

I have tried both the methods and neither worked unfortunalty :(

Method 1

Result = CStr(CDbl("1.65") - CDbl("1"))      This returned the 6.999999

Method 2

Returned this error:

error.JPG
0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
Rgonzo1971Commented:
Hi,

pls try

Result = FormatNumber(CDbl("1.067") - CDbl("1"), 3)
0
 
MacroShadowCommented:
Turns out CDec isn't supported in Asp.

Try this monstrous code, it may work (I've never used ASP.
Int(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value) + (Right(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value,Len(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value)-InStrRev(Transaction.Screen.Item("NONMREVIEW")Fields.Item(PREVQTY").Value,".")+1)) - Transaction.Screen.Item("NONMREVIEW")Fields.Item(QTY").Value

Open in new window

0
 
SmashAndGrabAuthor Commented:
perfect!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now