Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Math.Round Function is incorrect

Posted on 2009-06-30
7
Medium Priority
?
467 Views
Last Modified: 2013-11-26
If I get a value from my positiondatalocation = 94.8096 and I want to round up, the MS math.round function doesn;t work correctly it will give me an 94.809  for roundB

any suggestion? I would think the roundb would equal 94.810
Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)

Open in new window

0
Comment
Question by:cmdolcet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24747998
That works correctly for me, can you post more of your code or how you are seeing that it is 94.809 instead of 94.81 which is what I get.
0
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24748016
If you always want it to round up, even when you have 94.8093 for example, then try something like this instead.
roundB = Math.Ceiling(roundB * 1000) / 1000

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748018
Your thinking is correct. This is what it outputs for me:

?Math.Round(94.8096, 3)
94.81

How certain are you that the positiondatalocation constains what you think it contains? Can you show more of your code? Can you do a Debug.WriteLine just before your rounding statement?
0
Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

 

Author Comment

by:cmdolcet
ID: 24748059
The code is below
   positiondatalocation = CDbl(positiondata)
        Dim roundA As Double = positiondatalocation
        roundA = Math.Round(roundA, 3)
        ''If posStageloop = 0 Then
        'posCalcStage = positiondatalocation - CSng(STRdata)
        'positiondatalocation = positiondatalocation - posCalcStage
        Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''posStageloop = posStageloop + 1
        ''Else
 
        ''positiondatalocation = (positiondatalocation - intBAIoffset + StageValueOffset) '+ 0.03
        ''reports stage position each time the timer is fired in the status window.
        ''Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''End If
        '********************************************************************************************
 
 
 
        '**********************Sample loop for 1/2 inch indicator**************************************
        'Reads position of indicator with reference to the stage and the movement
        If _certificationflag = True Or _CertProgram = True Then
            If strTestType = "Indicator 1/2" Then
                If _BadResponse = True Then
                    _BadResponse = False
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Failed...."
                    IMGCetificationPass.Visible = False
                    IMGCetificationFail.Visible = True
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    TmrValidateIndicator.Enabled = True
                Else
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Passed...."
                    IMGCetificationPass.Visible = True
                    IMGCetificationFail.Visible = False
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    Me.lblStatusLabel.Text = String.Empty
 
                    _ActualStageMovPosition.Add(roundA)
                    _PositionReadingArraylistValidated.Add(roundA)
                    _PositionReadingArraylistValidated(intarrayindex) = roundA
                    positiondatalocation = roundA - absStageMove
                    Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)
                    absmoveposition = positiondatalocation
                    Baicomm.BAIAbsoluteStageMovement(1)

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748183
thanks for the code. Can you also answer the other question, about what is actually inside positiondata by adding a Debug.WriteLine just before the Math.Round? And one after would be nice, too. And show us what the Output window has:

positiondatalocation = CDbl(positiondata)
Debug.WriteLine("before: {0}", positiondatalocation)
Dim roundA As Double = positiondatalocation
Debug.WriteLine("after: {0}", roundA)
roundA = Math.Round(roundA, 3)

Open in new window

0
 

Author Comment

by:cmdolcet
ID: 24748349
abel I will need a couple of minutes to test your suggestion, please hold
thanks
0
 
LVL 60

Accepted Solution

by:
Kevin Cross earned 2000 total points
ID: 24748563
Note also that you are doing this:

positiondatalocation = roundA - absStageMove '<<<< changing position data value
Dim roundB As Double = positiondatalocation
roundB = Math.Round(roundB, 3)

So you will want to check what the value of positiondatalocation is after the subtraction.  Also, to note there is that the subtraction is occuring on the already rounded value; therefore, if you started with 94.81 you are subtracting from there and not 94.8096 so it might just be a coincidence that absStageMove is 0.001.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article describes relatively difficult and non-obvious issues that are likely to arise when creating COM class in Visual Studio and deploying it by professional MSI-authoring tools. It is assumed that the reader is already familiar with the cla…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
In this video you will find out how to export Office 365 mailboxes using the built in eDiscovery tool. Bear in mind that although this method might be useful in some cases, using PST files as Office 365 backup is troublesome in a long run (more on t…
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …

604 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question