Solved

Math.Round Function is incorrect

Posted on 2009-06-30
7
460 Views
Last Modified: 2013-11-26
If I get a value from my positiondatalocation = 94.8096 and I want to round up, the MS math.round function doesn;t work correctly it will give me an 94.809  for roundB

any suggestion? I would think the roundb would equal 94.810
Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)

Open in new window

0
Comment
Question by:cmdolcet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24747998
That works correctly for me, can you post more of your code or how you are seeing that it is 94.809 instead of 94.81 which is what I get.
0
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24748016
If you always want it to round up, even when you have 94.8093 for example, then try something like this instead.
roundB = Math.Ceiling(roundB * 1000) / 1000

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748018
Your thinking is correct. This is what it outputs for me:

?Math.Round(94.8096, 3)
94.81

How certain are you that the positiondatalocation constains what you think it contains? Can you show more of your code? Can you do a Debug.WriteLine just before your rounding statement?
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:cmdolcet
ID: 24748059
The code is below
   positiondatalocation = CDbl(positiondata)
        Dim roundA As Double = positiondatalocation
        roundA = Math.Round(roundA, 3)
        ''If posStageloop = 0 Then
        'posCalcStage = positiondatalocation - CSng(STRdata)
        'positiondatalocation = positiondatalocation - posCalcStage
        Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''posStageloop = posStageloop + 1
        ''Else
 
        ''positiondatalocation = (positiondatalocation - intBAIoffset + StageValueOffset) '+ 0.03
        ''reports stage position each time the timer is fired in the status window.
        ''Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''End If
        '********************************************************************************************
 
 
 
        '**********************Sample loop for 1/2 inch indicator**************************************
        'Reads position of indicator with reference to the stage and the movement
        If _certificationflag = True Or _CertProgram = True Then
            If strTestType = "Indicator 1/2" Then
                If _BadResponse = True Then
                    _BadResponse = False
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Failed...."
                    IMGCetificationPass.Visible = False
                    IMGCetificationFail.Visible = True
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    TmrValidateIndicator.Enabled = True
                Else
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Passed...."
                    IMGCetificationPass.Visible = True
                    IMGCetificationFail.Visible = False
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    Me.lblStatusLabel.Text = String.Empty
 
                    _ActualStageMovPosition.Add(roundA)
                    _PositionReadingArraylistValidated.Add(roundA)
                    _PositionReadingArraylistValidated(intarrayindex) = roundA
                    positiondatalocation = roundA - absStageMove
                    Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)
                    absmoveposition = positiondatalocation
                    Baicomm.BAIAbsoluteStageMovement(1)

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748183
thanks for the code. Can you also answer the other question, about what is actually inside positiondata by adding a Debug.WriteLine just before the Math.Round? And one after would be nice, too. And show us what the Output window has:

positiondatalocation = CDbl(positiondata)
Debug.WriteLine("before: {0}", positiondatalocation)
Dim roundA As Double = positiondatalocation
Debug.WriteLine("after: {0}", roundA)
roundA = Math.Round(roundA, 3)

Open in new window

0
 

Author Comment

by:cmdolcet
ID: 24748349
abel I will need a couple of minutes to test your suggestion, please hold
thanks
0
 
LVL 60

Accepted Solution

by:
Kevin Cross earned 500 total points
ID: 24748563
Note also that you are doing this:

positiondatalocation = roundA - absStageMove '<<<< changing position data value
Dim roundB As Double = positiondatalocation
roundB = Math.Round(roundB, 3)

So you will want to check what the value of positiondatalocation is after the subtraction.  Also, to note there is that the subtraction is occuring on the already rounded value; therefore, if you started with 94.81 you are subtracting from there and not 94.8096 so it might just be a coincidence that absStageMove is 0.001.
0

Featured Post

Get Actionable Data from Your Monitoring Solution

Your communication platform is only as good as the relevance of the information you send. Ensure your alerts get to the right people every time with actionable responses. Create escalation rules that ensure everyone follows the process and nothing is left to chance.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Many of us here at EE write code. Many of us write exceptional code; just as many of us write exception-prone code. As we all should know, exceptions are a mechanism for handling errors which are typically out of our control. From database errors, t…
Today I had a very interesting conundrum that had to get solved quickly. Needless to say, it wasn't resolved quickly because when we needed it we were very rushed, but as soon as the conference call was over and I took a step back I saw the correct …
In this video, viewers will be given step by step instructions on adjusting mouse, pointer and cursor visibility in Microsoft Windows 10. The video seeks to educate those who are struggling with the new Windows 10 Graphical User Interface. Change Cu…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…

695 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question