Fudging a rounding issue

Posted on 2011-05-09
Last Modified: 2012-05-11
I have an existing system that stores a measurement as an integer of 32th of an inch (1 in = 32).  The numbers are only valid for these increments.  I need to be able to convert this value to its millimeter equivalent and back.  The only allowed values for millimeters are whole numbers.  If I use a simple factor of 32/25.4 and then round the result I get a consistent mm.  But then I need to be able to convert back to inches and get the same value.  The problem is that due to rounding, the result may have been rounded up or down to the next valid integer.  And then if I convert back it may not have the original value.  The factor of 32/25.4 = 1.25984252.

We could live with a change, except for specific common values (mainly whole inch or at least foot values).  For example 3 feet or 36 inches = 2688, which when converted to mm = 914 (913.6 rounded up).  The problem is that when converting 914 back (914 * 1.25984252) we get 2689 (3' 1/32"), not the original 2688.  While I can see this is inevitable, given that there are more 32th per inch then millimeters.  I am looking for a way to adjust the result to avoid some common values.

When I graph the rounding issue, I see a pattern that makes me think I can recreate it on the reverse conversion enough to handle common values.

 Repetitive Rounding Error
Is it possible to devise a fudge factor that can keep the whole inch values from being effect by the rounding issue? At least for the first 20'?  As the issue is predictable, it should be possible to devise a way to compensate enough to always be able to convert from whole inches to mm and back to there original stored values.  I just can't seem to get something to work in all cases.

Here are some standard measurements that I need to have reproducible results for:

Measurement      Stored      mm      rounded      Refactored      Rounded      Delta
1 3/8"      44      35      35      44.0945      44      0
1 3/4"      56      44.45      44      55.4331      55      1
2"      64      51      51      64.2520      64      0
3'      1152      914.40      914      1151.4961      1151      1
7'      2688      2133.60      2134      2688.5039      2689      -1
3'6"      1344      1066.80      1067      1344.2520      1344      0
4'      1536      1219.20      1219      1535.7480      1536      0
4'6"      1728      1371.60      1372      1728.5039      1729      -1
5'      1920      1524.00      1524      1920.0000      1920      0
6'      2304      1828.80      1829      2304.2520      2304      0
6'8"      2560      2032.00      2032      2560.0000      2560      0
6'10"      2624      2082.80      2083      2624.2520      2624      0
7'      2688      2133.60      2134      2688.5039      2689      -1
7'2"      2752      2184.40      2184      2751.4961      2751      1
7'6"      2880      2286.00      2286      2880.0000      2880      0
7'10"      3008      2387.60      2388      3008.5039      3009      -1
8'      3072      2438.40      2438      3071.4961      3071      1
8'6"      3264      2590.80      2591      3264.2520      3264      0
9'      3456      2743.20      2743      3455.7480      3456      0
10'      3840      3048.00      3048      3840.0000      3840      0

I need the delta to be zero for these values.

Question by:GNiessen
    LVL 7

    Expert Comment

    Why not just relable the millimeter to micrometers.  That way you can just multiply everything by 100 to get your final answer.  This way you fudge having 1.00 millimeters to work with instead of just 1.
    LVL 17
    I would store a single value in whichever measurement system you choose and so make the calculation only in a single direction each time. So if you choose to store all your values as micrometers as bill30 suggests, then you will always have a consistent value when you convert.
    You only need to display whole numbers of millimeters, but you can store at any size you like.

    If you only need consistency up to 20 inches then you could just store the full list of conversions in a lookup table, there are only about 500 values if using whole millimeter values.

    Author Comment

    To maintain the existing capabilities, I am unable to change how the inches are stored.  And I would prefer a calculation as it should be faster then a lookup.  And then I would need the conversion class to have a database connection.  

    And I still have to devise a method to generate the stored values.  And do I have to have two lookups for converting to and from millimeters, since there are more 32nds then mm?
    LVL 26

    Expert Comment

    Is this real data?  Are you measuring little parts and keeping track of the inventory?

    If so, can you store both numbers?

    If you care about accuracy, you really can't use this system.

    If you measure in mm and convert to 32nds, 40% of the measurements will be ambiguous.

    Exact   Exact          Rounded
     mm     inches          32nd
    1	0.0313  	 1
    2	0.0625  	 2 |
    3	0.0938  	 2 |
    4	0.1250  	 3
    5	0.1563  	 4
    6	0.1875  	 5
    7	0.2188  	 6 |
    8	0.2500  	 6 |
    9	0.2813  	 7
    10	0.3125  	 8
    11	0.3438  	 9
    12	0.3750  	10 |
    13	0.4063  	10 |
    14	0.4375  	11
    15	0.4688  	12
    16	0.5000  	13 |
    17	0.5313  	13 |
    18	0.5625  	14
    19	0.5938  	15
    20	0.6250  	16

    Open in new window

    LVL 7

    Expert Comment

    1. A table lookup of this kind will not be slow at all.  
    2. You say you are fine with a calculation, you can add the x100 calculation limited to this project, and enter micrometers instead of millimeters as stated before.
    3. If none of the answers above suit you just try changing the database value to a decimal instead of an integer, giving you enough digits to be accurate.
    LVL 37

    Expert Comment

    Since the right side of your mapping is smaller than the left (32 32nds in an inch, but 25mm) it is impossible to map correctly back and forth. Since you cannot change how the inches are stored and since the inches are the more precise, then stop converting back and forth (always a bad idea, try it with Google translator!)

    I agree with RobinD. Keep the inches stored as they are and every time you need a mm convert from inches. The only other way would be to store the millimeters with one decimal digit and divide by 10 before use each time.

    LVL 37

    Expert Comment

    But, if you insist, then this is the best option (by far, in my opinion)

    Convert to 16ths of an inch on the way back then store it as 32nds. So basically just round to the nearest even integer. This way you have a simple, easy, consistent solution.

    Since you know the precision will be off by at least that much anyway, now it's standardized across the whole thing and you don't need to worry about finding patterns (your patterns aren't going to repeat at a useful period, you'll need more rounding and have potential for error in higher numbers)

    Rounding to nearest even number of 32nds is robust and easy and as accurate as any other method so I would highly recommend doing it that way.

    Author Comment

    I cannot change the stored value without effecting the existing system.  And I need to support constraints coded against the stored values.

    As long as the resulting converted value is not greater, and only the whole inches are of concern, then using a conversion of 32/25.4 and a fudge factor of .02 works for all but 8 instances in the first 42 feet of measurements.  If I use -.02 for those they work also.  Doing that means that any mm value can be stored and the resulting value matches the original mm.  There is a loss when converting a inches value is about 20% of the cases where the stored value is 1 less or 1/32th of an inch.  But I think I can live with that as 32 can't go into 25.4 without some loss.

    I was hoping someone would figure out a sign wave or ordinal pattern that would allow me to compensate for the error in a consistent way and not use a hard coded list to avoid conflicts that effect the system.  After seeing the graph of the rounding error, it just seemed like something was possible.
    LVL 37

    Expert Comment

    I was hoping someone would figure out a sign wave or ordinal pattern that
    It's not a sine wave, it's a linear function. Each time you convert 1/32 you have an extra .7656ish that rolls over. If it was .6 for example, you'd see this pattern in the decimal.
    .6, .2, .8, .4, 0, .6 repeating. Problem is that you still lose data because some mm numbers need to go to different inch values. Another problem is that your pattern wouldn't have the same period every time. Sometimes it would 'wrap' in 24 mm, sometimes n 25.

    32 can't go into 25.4 without some loss.

    Exactly. The right way to do it is to absorb the loss in a fair way. See, no matter what value you get back for the inches, you know it's +/- 1/32 of what it is supposed to be. Remember significant figures from Science or Math classes in High School? Same idea here.

    No method you find will give you any better precision than rounding to the nearest 16th of an inch when converting back without changing the structure so you can 'remember' what it came from. Rounding to the nearest 16th will give consistent results (positive and negative error will be equal). It also won't freak out any mathematicians who see your work later.
    LVL 17
    I tried something similar to remove the tax from prices to get the original pre-tax price. The conclusion was that it cannot be done. The information has been removed from the value when the rounding was done and it cannot be retrieved accurately. For consistent results over a known range of values a look-up table (even one stored in an array inside your module and not in a database) will give the best result.
    LVL 7

    Expert Comment

    GNiessen you keep saying you dont want to affect the entire system.  If you can add an equation to part of the system, you can change the milli to micro for part of the system, or use a table lookup for part of the system.  
    LVL 37

    Expert Comment

    But if you choose to round some numbers to the closest 32nd and some to the closest inch or half inch, then it's bad math and you'll have bias in your error. When you 'fudge' the ones close to whole inches to whole inches what you are really doing is selectively rounding some numbers to the nearest 16th. If you want some rounded more than to the nearest 16th, then you should do it on all of them.

    Author Comment

    I am not trying to map 32 into 25.4 as much as map the 25.4 into the 32 in a consistent way.  If I round to the nearest 16th, then I loose some of the 25.4.  I need to be able to support any mm measurement.  And match mm to inches with some consistency.  

    I think you may have something in your ".6, .2, .8, .4, 0, .6 repeating".  But it is more like there are two overlapping patterns.  If I can model them, then I can map them to a fudge factor that gives me the results i need.
    LVL 37

    Accepted Solution

    The 'two overlapping patterns' are because it doesn't hit 0. For example, if the overage was .61 then you would see .61, .22, .8., .44, .05, .66, .27, etc.
    So there would be a quick loop of about 5 as before, but then it would slowly increase and then that slow increase would loop as well (as you see in your data). But it wouldn't always be 5. The problem is that your decimal may not be that simple. .61 would have two loops. One with a period of a bit more than 5 and the other 10 and a combined loop period of 100. Yours would have an even longer one and it would depend on where you put it. If you are using 25.4 mm per inch then So your pattern is messy and is itself subject to error, especially with the fact that decimals are stored in binary on the computer anyway.

    What are you using anyway? 25.4 mm per inch? If so that yours wraps every 160/32 or every 5 inches.

    I need to be able to support any mm measurement.  And match mm to inches with some consistency.  
    There are only two ways to do this:
    1. Round to the nearest 1/32 and accept the fact that you'll miss some even though some of them are nice whole numbers.
    2. Round to the nearest 1/16 and accept the fact that you lost a little more precision.

    Again, if you round to 1/32 and then round the ones close to a 1 or 1/2 to the nearest 1/16, then you can't have consistency and you will have biased error. However, if you insist on doing it this way (please don't) then just convert the mm to inches and if it's within 1/32 of a target number, round it that way.

    Author Closing Comment

    Not a plug and play solution, but I should be able to make it work.  Thanks.

    Featured Post

    Find Ransomware Secrets With All-Source Analysis

    Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    Functions 7 45
    Graph 2 45
    x2 – x2 = x2 – x2 conundrum 12 57
    Triangles - computing angles 7 25
    A Guide to the PMT, FV, IPMT and PPMT Functions In MS Excel we have the PMT, FV, IPMT and PPMT functions, which do a fantastic job for interest rate calculations.  But what if you don't have Excel ? This article is for programmers looking to re…
    How to Win a Jar of Candy Corn: A Scientific Approach! I love mathematics. If you love mathematics also, you may enjoy this tip on how to use math to win your own jar of candy corn and to impress your friends. As I said, I love math, but I gu…
    Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
    This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor ( If you're looking for how to monitor bandwidth using netflow or packet s…

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now