Solved

Math.Round Function is incorrect

Posted on 2009-06-30
7
454 Views
Last Modified: 2013-11-26
If I get a value from my positiondatalocation = 94.8096 and I want to round up, the MS math.round function doesn;t work correctly it will give me an 94.809  for roundB

any suggestion? I would think the roundb would equal 94.810
Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)

Open in new window

0
Comment
Question by:cmdolcet
  • 3
  • 2
  • 2
7 Comments
 
LVL 59

Expert Comment

by:Kevin Cross
ID: 24747998
That works correctly for me, can you post more of your code or how you are seeing that it is 94.809 instead of 94.81 which is what I get.
0
 
LVL 59

Expert Comment

by:Kevin Cross
ID: 24748016
If you always want it to round up, even when you have 94.8093 for example, then try something like this instead.
roundB = Math.Ceiling(roundB * 1000) / 1000

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748018
Your thinking is correct. This is what it outputs for me:

?Math.Round(94.8096, 3)
94.81

How certain are you that the positiondatalocation constains what you think it contains? Can you show more of your code? Can you do a Debug.WriteLine just before your rounding statement?
0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 

Author Comment

by:cmdolcet
ID: 24748059
The code is below
   positiondatalocation = CDbl(positiondata)
        Dim roundA As Double = positiondatalocation
        roundA = Math.Round(roundA, 3)
        ''If posStageloop = 0 Then
        'posCalcStage = positiondatalocation - CSng(STRdata)
        'positiondatalocation = positiondatalocation - posCalcStage
        Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''posStageloop = posStageloop + 1
        ''Else
 
        ''positiondatalocation = (positiondatalocation - intBAIoffset + StageValueOffset) '+ 0.03
        ''reports stage position each time the timer is fired in the status window.
        ''Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''End If
        '********************************************************************************************
 
 
 
        '**********************Sample loop for 1/2 inch indicator**************************************
        'Reads position of indicator with reference to the stage and the movement
        If _certificationflag = True Or _CertProgram = True Then
            If strTestType = "Indicator 1/2" Then
                If _BadResponse = True Then
                    _BadResponse = False
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Failed...."
                    IMGCetificationPass.Visible = False
                    IMGCetificationFail.Visible = True
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    TmrValidateIndicator.Enabled = True
                Else
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Passed...."
                    IMGCetificationPass.Visible = True
                    IMGCetificationFail.Visible = False
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    Me.lblStatusLabel.Text = String.Empty
 
                    _ActualStageMovPosition.Add(roundA)
                    _PositionReadingArraylistValidated.Add(roundA)
                    _PositionReadingArraylistValidated(intarrayindex) = roundA
                    positiondatalocation = roundA - absStageMove
                    Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)
                    absmoveposition = positiondatalocation
                    Baicomm.BAIAbsoluteStageMovement(1)

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748183
thanks for the code. Can you also answer the other question, about what is actually inside positiondata by adding a Debug.WriteLine just before the Math.Round? And one after would be nice, too. And show us what the Output window has:

positiondatalocation = CDbl(positiondata)
Debug.WriteLine("before: {0}", positiondatalocation)
Dim roundA As Double = positiondatalocation
Debug.WriteLine("after: {0}", roundA)
roundA = Math.Round(roundA, 3)

Open in new window

0
 

Author Comment

by:cmdolcet
ID: 24748349
abel I will need a couple of minutes to test your suggestion, please hold
thanks
0
 
LVL 59

Accepted Solution

by:
Kevin Cross earned 500 total points
ID: 24748563
Note also that you are doing this:

positiondatalocation = roundA - absStageMove '<<<< changing position data value
Dim roundB As Double = positiondatalocation
roundB = Math.Round(roundB, 3)

So you will want to check what the value of positiondatalocation is after the subtraction.  Also, to note there is that the subtraction is occuring on the already rounded value; therefore, if you started with 94.81 you are subtracting from there and not 94.8096 so it might just be a coincidence that absStageMove is 0.001.
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

More often than not, we developers are confronted with a need: a need to make some kind of magic happen via code. Whether it is for a client, for the boss, or for our own personal projects, the need must be satisfied. Most of the time, the Framework…
A long time ago (May 2011), I have written an article showing you how to create a DLL using Visual Studio 2005 to be hosted in SQL Server 2005. That was valid at that time and it is still valid if you are still using these versions. You can still re…
Established in 1997, Technology Architects has become one of the most reputable technology solutions companies in the country. TA have been providing businesses with cost effective state-of-the-art solutions and unparalleled service that is designed…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question