?
Solved

Math.Round Function is incorrect

Posted on 2009-06-30
7
Medium Priority
?
463 Views
Last Modified: 2013-11-26
If I get a value from my positiondatalocation = 94.8096 and I want to round up, the MS math.round function doesn;t work correctly it will give me an 94.809  for roundB

any suggestion? I would think the roundb would equal 94.810
Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)

Open in new window

0
Comment
Question by:cmdolcet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24747998
That works correctly for me, can you post more of your code or how you are seeing that it is 94.809 instead of 94.81 which is what I get.
0
 
LVL 60

Expert Comment

by:Kevin Cross
ID: 24748016
If you always want it to round up, even when you have 94.8093 for example, then try something like this instead.
roundB = Math.Ceiling(roundB * 1000) / 1000

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748018
Your thinking is correct. This is what it outputs for me:

?Math.Round(94.8096, 3)
94.81

How certain are you that the positiondatalocation constains what you think it contains? Can you show more of your code? Can you do a Debug.WriteLine just before your rounding statement?
0
Setting up LaraDock for Laravel

Learn how to set up LaraDock in a Laravel project - LaraDock gives us an easy way to run a Laravel application using Docker in a single command.

 

Author Comment

by:cmdolcet
ID: 24748059
The code is below
   positiondatalocation = CDbl(positiondata)
        Dim roundA As Double = positiondatalocation
        roundA = Math.Round(roundA, 3)
        ''If posStageloop = 0 Then
        'posCalcStage = positiondatalocation - CSng(STRdata)
        'positiondatalocation = positiondatalocation - posCalcStage
        Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''posStageloop = posStageloop + 1
        ''Else
 
        ''positiondatalocation = (positiondatalocation - intBAIoffset + StageValueOffset) '+ 0.03
        ''reports stage position each time the timer is fired in the status window.
        ''Me.StatusPositionEncoder.Text = "Stage Position:...." & positiondatalocation
        ''End If
        '********************************************************************************************
 
 
 
        '**********************Sample loop for 1/2 inch indicator**************************************
        'Reads position of indicator with reference to the stage and the movement
        If _certificationflag = True Or _CertProgram = True Then
            If strTestType = "Indicator 1/2" Then
                If _BadResponse = True Then
                    _BadResponse = False
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Failed...."
                    IMGCetificationPass.Visible = False
                    IMGCetificationFail.Visible = True
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    TmrValidateIndicator.Enabled = True
                Else
                    Me.lblStatusLabel.Text = String.Empty
                    Me.lblCertStatusLabel.Text = "Position Checked Passed...."
                    IMGCetificationPass.Visible = True
                    IMGCetificationFail.Visible = False
                    ReadingStatusBad.Visible = False
                    ReadingStatusOK.Visible = False
                    Me.lblStatusLabel.Text = String.Empty
 
                    _ActualStageMovPosition.Add(roundA)
                    _PositionReadingArraylistValidated.Add(roundA)
                    _PositionReadingArraylistValidated(intarrayindex) = roundA
                    positiondatalocation = roundA - absStageMove
                    Dim roundB As Double = positiondatalocation
                    roundB = Math.Round(roundB, 3)
                    absmoveposition = positiondatalocation
                    Baicomm.BAIAbsoluteStageMovement(1)

Open in new window

0
 
LVL 39

Expert Comment

by:abel
ID: 24748183
thanks for the code. Can you also answer the other question, about what is actually inside positiondata by adding a Debug.WriteLine just before the Math.Round? And one after would be nice, too. And show us what the Output window has:

positiondatalocation = CDbl(positiondata)
Debug.WriteLine("before: {0}", positiondatalocation)
Dim roundA As Double = positiondatalocation
Debug.WriteLine("after: {0}", roundA)
roundA = Math.Round(roundA, 3)

Open in new window

0
 

Author Comment

by:cmdolcet
ID: 24748349
abel I will need a couple of minutes to test your suggestion, please hold
thanks
0
 
LVL 60

Accepted Solution

by:
Kevin Cross earned 2000 total points
ID: 24748563
Note also that you are doing this:

positiondatalocation = roundA - absStageMove '<<<< changing position data value
Dim roundB As Double = positiondatalocation
roundB = Math.Round(roundB, 3)

So you will want to check what the value of positiondatalocation is after the subtraction.  Also, to note there is that the subtraction is occuring on the already rounded value; therefore, if you started with 94.81 you are subtracting from there and not 94.8096 so it might just be a coincidence that absStageMove is 0.001.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
This article shows how to deploy dynamic backgrounds to computers depending on the aspect ratio of display
If you’ve ever visited a web page and noticed a cool font that you really liked the look of, but couldn’t figure out which font it was so that you could use it for your own work, then this video is for you! In this Micro Tutorial, you'll learn yo…
Do you want to know how to make a graph with Microsoft Access? First, create a query with the data for the chart. Then make a blank form and add a chart control. This video also shows how to change what data is displayed on the graph as well as form…
Suggested Courses

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question