Solved

Posted on 2011-05-09

I have an existing system that stores a measurement as an integer of 32th of an inch (1 in = 32). The numbers are only valid for these increments. I need to be able to convert this value to its millimeter equivalent and back. The only allowed values for millimeters are whole numbers. If I use a simple factor of 32/25.4 and then round the result I get a consistent mm. But then I need to be able to convert back to inches and get the same value. The problem is that due to rounding, the result may have been rounded up or down to the next valid integer. And then if I convert back it may not have the original value. The factor of 32/25.4 = 1.25984252.

We could live with a change, except for specific common values (mainly whole inch or at least foot values). For example 3 feet or 36 inches = 2688, which when converted to mm = 914 (913.6 rounded up). The problem is that when converting 914 back (914 * 1.25984252) we get 2689 (3' 1/32"), not the original 2688. While I can see this is inevitable, given that there are more 32th per inch then millimeters. I am looking for a way to adjust the result to avoid some common values.

When I graph the rounding issue, I see a pattern that makes me think I can recreate it on the reverse conversion enough to handle common values.

Is it possible to devise a fudge factor that can keep the whole inch values from being effect by the rounding issue? At least for the first 20'? As the issue is predictable, it should be possible to devise a way to compensate enough to always be able to convert from whole inches to mm and back to there original stored values. I just can't seem to get something to work in all cases.

Here are some standard measurements that I need to have reproducible results for:

Measurement Stored mm rounded Refactored Rounded Delta

1 3/8" 44 35 35 44.0945 44 0

1 3/4" 56 44.45 44 55.4331 55 1

2" 64 51 51 64.2520 64 0

3' 1152 914.40 914 1151.4961 1151 1

7' 2688 2133.60 2134 2688.5039 2689 -1

3'6" 1344 1066.80 1067 1344.2520 1344 0

4' 1536 1219.20 1219 1535.7480 1536 0

4'6" 1728 1371.60 1372 1728.5039 1729 -1

5' 1920 1524.00 1524 1920.0000 1920 0

6' 2304 1828.80 1829 2304.2520 2304 0

6'8" 2560 2032.00 2032 2560.0000 2560 0

6'10" 2624 2082.80 2083 2624.2520 2624 0

7' 2688 2133.60 2134 2688.5039 2689 -1

7'2" 2752 2184.40 2184 2751.4961 2751 1

7'6" 2880 2286.00 2286 2880.0000 2880 0

7'10" 3008 2387.60 2388 3008.5039 3009 -1

8' 3072 2438.40 2438 3071.4961 3071 1

8'6" 3264 2590.80 2591 3264.2520 3264 0

9' 3456 2743.20 2743 3455.7480 3456 0

10' 3840 3048.00 3048 3840.0000 3840 0

I need the delta to be zero for these values.

We could live with a change, except for specific common values (mainly whole inch or at least foot values). For example 3 feet or 36 inches = 2688, which when converted to mm = 914 (913.6 rounded up). The problem is that when converting 914 back (914 * 1.25984252) we get 2689 (3' 1/32"), not the original 2688. While I can see this is inevitable, given that there are more 32th per inch then millimeters. I am looking for a way to adjust the result to avoid some common values.

When I graph the rounding issue, I see a pattern that makes me think I can recreate it on the reverse conversion enough to handle common values.

Is it possible to devise a fudge factor that can keep the whole inch values from being effect by the rounding issue? At least for the first 20'? As the issue is predictable, it should be possible to devise a way to compensate enough to always be able to convert from whole inches to mm and back to there original stored values. I just can't seem to get something to work in all cases.

Here are some standard measurements that I need to have reproducible results for:

Measurement Stored mm rounded Refactored Rounded Delta

1 3/8" 44 35 35 44.0945 44 0

1 3/4" 56 44.45 44 55.4331 55 1

2" 64 51 51 64.2520 64 0

3' 1152 914.40 914 1151.4961 1151 1

7' 2688 2133.60 2134 2688.5039 2689 -1

3'6" 1344 1066.80 1067 1344.2520 1344 0

4' 1536 1219.20 1219 1535.7480 1536 0

4'6" 1728 1371.60 1372 1728.5039 1729 -1

5' 1920 1524.00 1524 1920.0000 1920 0

6' 2304 1828.80 1829 2304.2520 2304 0

6'8" 2560 2032.00 2032 2560.0000 2560 0

6'10" 2624 2082.80 2083 2624.2520 2624 0

7' 2688 2133.60 2134 2688.5039 2689 -1

7'2" 2752 2184.40 2184 2751.4961 2751 1

7'6" 2880 2286.00 2286 2880.0000 2880 0

7'10" 3008 2387.60 2388 3008.5039 3009 -1

8' 3072 2438.40 2438 3071.4961 3071 1

8'6" 3264 2590.80 2591 3264.2520 3264 0

9' 3456 2743.20 2743 3455.7480 3456 0

10' 3840 3048.00 3048 3840.0000 3840 0

I need the delta to be zero for these values.

15 Comments

You only need to display whole numbers of millimeters, but you can store at any size you like.

If you only need consistency up to 20 inches then you could just store the full list of conversions in a lookup table, there are only about 500 values if using whole millimeter values.

And I still have to devise a method to generate the stored values. And do I have to have two lookups for converting to and from millimeters, since there are more 32nds then mm?

If so, can you store both numbers?

If you care about accuracy, you really can't use this system.

If you measure in mm and convert to 32nds, 40% of the measurements will be ambiguous.

```
Exact Exact Rounded
mm inches 32nd
1 0.0313 1
2 0.0625 2 |
3 0.0938 2 |
4 0.1250 3
5 0.1563 4
6 0.1875 5
7 0.2188 6 |
8 0.2500 6 |
9 0.2813 7
10 0.3125 8
11 0.3438 9
12 0.3750 10 |
13 0.4063 10 |
14 0.4375 11
15 0.4688 12
16 0.5000 13 |
17 0.5313 13 |
18 0.5625 14
19 0.5938 15
20 0.6250 16
```

2. You say you are fine with a calculation, you can add the x100 calculation limited to this project, and enter micrometers instead of millimeters as stated before.

3. If none of the answers above suit you just try changing the database value to a decimal instead of an integer, giving you enough digits to be accurate.

I agree with RobinD. Keep the inches stored as they are and every time you need a mm convert from inches. The only other way would be to store the millimeters with one decimal digit and divide by 10 before use each time.

Convert to 16ths of an inch on the way back then store it as 32nds. So basically just round to the nearest even integer. This way you have a simple, easy, consistent solution.

Since you know the precision will be off by at least that much anyway, now it's standardized across the whole thing and you don't need to worry about finding patterns (your patterns aren't going to repeat at a useful period, you'll need more rounding and have potential for error in higher numbers)

Rounding to nearest even number of 32nds is robust and easy and as accurate as any other method so I would highly recommend doing it that way.

As long as the resulting converted value is not greater, and only the whole inches are of concern, then using a conversion of 32/25.4 and a fudge factor of .02 works for all but 8 instances in the first 42 feet of measurements. If I use -.02 for those they work also. Doing that means that any mm value can be stored and the resulting value matches the original mm. There is a loss when converting a inches value is about 20% of the cases where the stored value is 1 less or 1/32th of an inch. But I think I can live with that as 32 can't go into 25.4 without some loss.

I was hoping someone would figure out a sign wave or ordinal pattern that would allow me to compensate for the error in a consistent way and not use a hard coded list to avoid conflicts that effect the system. After seeing the graph of the rounding error, it just seemed like something was possible.

It's not a sine wave, it's a linear function. Each time you convert 1/32 you have an extra .7656ish that rolls over. If it was .6 for example, you'd see this pattern in the decimal.

.6, .2, .8, .4, 0, .6 repeating. Problem is that you still lose data because some mm numbers need to go to different inch values. Another problem is that your pattern wouldn't have the same period every time. Sometimes it would 'wrap' in 24 mm, sometimes n 25.

Exactly. The right way to do it is to absorb the loss in a fair way. See, no matter what value you get back for the inches, you know it's +/- 1/32 of what it is supposed to be. Remember significant figures from Science or Math classes in High School? Same idea here.

No method you find will give you any better precision than rounding to the nearest 16th of an inch when converting back without changing the structure so you can 'remember' what it came from. Rounding to the nearest 16th will give consistent results (positive and negative error will be equal). It also won't freak out any mathematicians who see your work later.

Tommy,

I think you may have something in your ".6, .2, .8, .4, 0, .6 repeating". But it is more like there are two overlapping patterns. If I can model them, then I can map them to a fudge factor that gives me the results i need.

So there would be a quick loop of about 5 as before, but then it would slowly increase and then that slow increase would loop as well (as you see in your data). But it wouldn't always be 5. The problem is that your decimal may not be that simple. .61 would have two loops. One with a period of a bit more than 5 and the other 10 and a combined loop period of 100. Yours would have an even longer one and it would depend on where you put it. If you are using 25.4 mm per inch then So your pattern is messy and is itself subject to error, especially with the fact that decimals are stored in binary on the computer anyway.

What are you using anyway? 25.4 mm per inch? If so that yours wraps every 160/32 or every 5 inches.

There are only two ways to do this:

1. Round to the nearest 1/32 and accept the fact that you'll miss some even though some of them are nice whole numbers.

2. Round to the nearest 1/16 and accept the fact that you lost a little more precision.

Again, if you round to 1/32 and then round the ones close to a 1 or 1/2 to the nearest 1/16, then you can't have consistency and you will have biased error. However, if you insist on doing it this way (please don't) then just convert the mm to inches and if it's within 1/32 of a target number, round it that way.

Title | # Comments | Views | Activity |
---|---|---|---|

Functions | 7 | 45 | |

Graph | 2 | 45 | |

x2 – x2 = x2 – x2 conundrum | 12 | 57 | |

Triangles - computing angles | 7 | 25 |

Join the community of 500,000 technology professionals and ask your questions.

Connect with top rated Experts

**20** Experts available now in Live!