Solved

Math.Round Function is incorrect

Posted on 2009-06-30
7
446 Views
Last Modified: 2013-11-26
If I get a value from my positiondatalocation = 94.8096 and I want to round up, the MS math.round function doesn;t work correctly it will give me an 94.809  for roundB

any suggestion? I would think the roundb would equal 94.810
Dim roundB As Double = positiondatalocation

                    roundB = Math.Round(roundB, 3)

Open in new window

0
Comment
Question by:cmdolcet
  • 3
  • 2
  • 2
7 Comments
 
LVL 59

Expert Comment

by:Kevin Cross
Comment Utility
That works correctly for me, can you post more of your code or how you are seeing that it is 94.809 instead of 94.81 which is what I get.
0
 
LVL 59

Expert Comment

by:Kevin Cross
Comment Utility
If you always want it to round up, even when you have 94.8093 for example, then try something like this instead.
roundB = Math.Ceiling(roundB * 1000) / 1000

Open in new window

0
 
LVL 39

Expert Comment

by:abel
Comment Utility
Your thinking is correct. This is what it outputs for me:

?Math.Round(94.8096, 3)
94.81

How certain are you that the positiondatalocation constains what you think it contains? Can you show more of your code? Can you do a Debug.WriteLine just before your rounding statement?
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 

Author Comment

by:cmdolcet
Comment Utility
The code is below
   positiondatalocation = CDbl(positiondata)

        Dim roundA As Double = positiondatalocation

        roundA = Math.Round(roundA, 3)

        ''If posStageloop = 0 Then

        'posCalcStage = positiondatalocation - CSng(STRdata)

        'positiondatalocation = positiondatalocation - posCalcStage

        Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation

        ''posStageloop = posStageloop + 1

        ''Else
 

        ''positiondatalocation = (positiondatalocation - intBAIoffset + StageValueOffset) '+ 0.03

        ''reports stage position each time the timer is fired in the status window.

        ''Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation

        ''End If

        '********************************************************************************************
 
 
 

        '**********************Sample loop for 1/2 inch indicator**************************************

        'Reads position of indicator with reference to the stage and the movement

        If _certificationflag = True Or _CertProgram = True Then

            If strTestType = "Indicator 1/2" Then

                If _BadResponse = True Then

                    _BadResponse = False

                    Me.lblStatusLabel.Text = String.Empty

                    Me.lblCertStatusLabel.Text = "Position Checked Failed...."

                    IMGCetificationPass.Visible = False

                    IMGCetificationFail.Visible = True

                    ReadingStatusBad.Visible = False

                    ReadingStatusOK.Visible = False

                    TmrValidateIndicator.Enabled = True

                Else

                    Me.lblStatusLabel.Text = String.Empty

                    Me.lblCertStatusLabel.Text = "Position Checked Passed...."

                    IMGCetificationPass.Visible = True

                    IMGCetificationFail.Visible = False

                    ReadingStatusBad.Visible = False

                    ReadingStatusOK.Visible = False

                    Me.lblStatusLabel.Text = String.Empty
 

                    _ActualStageMovPosition.Add(roundA)

                    _PositionReadingArraylistValidated.Add(roundA)

                    _PositionReadingArraylistValidated(intarrayindex) = roundA

                    positiondatalocation = roundA - absStageMove

                    Dim roundB As Double = positiondatalocation

                    roundB = Math.Round(roundB, 3)

                    absmoveposition = positiondatalocation

                    Baicomm.BAIAbsoluteStageMovement(1)

Open in new window

0
 
LVL 39

Expert Comment

by:abel
Comment Utility
thanks for the code. Can you also answer the other question, about what is actually inside positiondata by adding a Debug.WriteLine just before the Math.Round? And one after would be nice, too. And show us what the Output window has:

positiondatalocation = CDbl(positiondata)

Debug.WriteLine("before: {0}", positiondatalocation)

Dim roundA As Double = positiondatalocation

Debug.WriteLine("after: {0}", roundA)

roundA = Math.Round(roundA, 3)

Open in new window

0
 

Author Comment

by:cmdolcet
Comment Utility
abel I will need a couple of minutes to test your suggestion, please hold
thanks
0
 
LVL 59

Accepted Solution

by:
Kevin Cross earned 500 total points
Comment Utility
Note also that you are doing this:

positiondatalocation = roundA - absStageMove '<<<< changing position data value
Dim roundB As Double = positiondatalocation
roundB = Math.Round(roundB, 3)

So you will want to check what the value of positiondatalocation is after the subtraction.  Also, to note there is that the subtraction is occuring on the already rounded value; therefore, if you started with 94.81 you are subtracting from there and not 94.8096 so it might just be a coincidence that absStageMove is 0.001.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Parsing a CSV file is a task that we are confronted with regularly, and although there are a vast number of means to do this, as a newbie, the field can be confusing and the tools can seem complex. A simple solution to parsing a customized CSV fi…
The ECB site provides FX rates for major currencies since its inception in 1999 in the form of an XML feed. The files have the following format (reducted for brevity) (CODE) There are three files available HERE (http://www.ecb.europa.eu/stats/exch…
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now