GNiessen
asked on
Fudging a rounding issue
I have an existing system that stores a measurement as an integer of 32th of an inch (1 in = 32). The numbers are only valid for these increments. I need to be able to convert this value to its millimeter equivalent and back. The only allowed values for millimeters are whole numbers. If I use a simple factor of 32/25.4 and then round the result I get a consistent mm. But then I need to be able to convert back to inches and get the same value. The problem is that due to rounding, the result may have been rounded up or down to the next valid integer. And then if I convert back it may not have the original value. The factor of 32/25.4 = 1.25984252.
We could live with a change, except for specific common values (mainly whole inch or at least foot values). For example 3 feet or 36 inches = 2688, which when converted to mm = 914 (913.6 rounded up). The problem is that when converting 914 back (914 * 1.25984252) we get 2689 (3' 1/32"), not the original 2688. While I can see this is inevitable, given that there are more 32th per inch then millimeters. I am looking for a way to adjust the result to avoid some common values.
When I graph the rounding issue, I see a pattern that makes me think I can recreate it on the reverse conversion enough to handle common values.
Is it possible to devise a fudge factor that can keep the whole inch values from being effect by the rounding issue? At least for the first 20'? As the issue is predictable, it should be possible to devise a way to compensate enough to always be able to convert from whole inches to mm and back to there original stored values. I just can't seem to get something to work in all cases.
Here are some standard measurements that I need to have reproducible results for:
Measurement Stored mm rounded Refactored Rounded Delta
1 3/8" 44 35 35 44.0945 44 0
1 3/4" 56 44.45 44 55.4331 55 1
2" 64 51 51 64.2520 64 0
3' 1152 914.40 914 1151.4961 1151 1
7' 2688 2133.60 2134 2688.5039 2689 -1
3'6" 1344 1066.80 1067 1344.2520 1344 0
4' 1536 1219.20 1219 1535.7480 1536 0
4'6" 1728 1371.60 1372 1728.5039 1729 -1
5' 1920 1524.00 1524 1920.0000 1920 0
6' 2304 1828.80 1829 2304.2520 2304 0
6'8" 2560 2032.00 2032 2560.0000 2560 0
6'10" 2624 2082.80 2083 2624.2520 2624 0
7' 2688 2133.60 2134 2688.5039 2689 -1
7'2" 2752 2184.40 2184 2751.4961 2751 1
7'6" 2880 2286.00 2286 2880.0000 2880 0
7'10" 3008 2387.60 2388 3008.5039 3009 -1
8' 3072 2438.40 2438 3071.4961 3071 1
8'6" 3264 2590.80 2591 3264.2520 3264 0
9' 3456 2743.20 2743 3455.7480 3456 0
10' 3840 3048.00 3048 3840.0000 3840 0
I need the delta to be zero for these values.
We could live with a change, except for specific common values (mainly whole inch or at least foot values). For example 3 feet or 36 inches = 2688, which when converted to mm = 914 (913.6 rounded up). The problem is that when converting 914 back (914 * 1.25984252) we get 2689 (3' 1/32"), not the original 2688. While I can see this is inevitable, given that there are more 32th per inch then millimeters. I am looking for a way to adjust the result to avoid some common values.
When I graph the rounding issue, I see a pattern that makes me think I can recreate it on the reverse conversion enough to handle common values.
Is it possible to devise a fudge factor that can keep the whole inch values from being effect by the rounding issue? At least for the first 20'? As the issue is predictable, it should be possible to devise a way to compensate enough to always be able to convert from whole inches to mm and back to there original stored values. I just can't seem to get something to work in all cases.
Here are some standard measurements that I need to have reproducible results for:
Measurement Stored mm rounded Refactored Rounded Delta
1 3/8" 44 35 35 44.0945 44 0
1 3/4" 56 44.45 44 55.4331 55 1
2" 64 51 51 64.2520 64 0
3' 1152 914.40 914 1151.4961 1151 1
7' 2688 2133.60 2134 2688.5039 2689 -1
3'6" 1344 1066.80 1067 1344.2520 1344 0
4' 1536 1219.20 1219 1535.7480 1536 0
4'6" 1728 1371.60 1372 1728.5039 1729 -1
5' 1920 1524.00 1524 1920.0000 1920 0
6' 2304 1828.80 1829 2304.2520 2304 0
6'8" 2560 2032.00 2032 2560.0000 2560 0
6'10" 2624 2082.80 2083 2624.2520 2624 0
7' 2688 2133.60 2134 2688.5039 2689 -1
7'2" 2752 2184.40 2184 2751.4961 2751 1
7'6" 2880 2286.00 2286 2880.0000 2880 0
7'10" 3008 2387.60 2388 3008.5039 3009 -1
8' 3072 2438.40 2438 3071.4961 3071 1
8'6" 3264 2590.80 2591 3264.2520 3264 0
9' 3456 2743.20 2743 3455.7480 3456 0
10' 3840 3048.00 3048 3840.0000 3840 0
I need the delta to be zero for these values.
bill30
Why not just relable the millimeter to micrometers. That way you can just multiply everything by 100 to get your final answer. This way you fudge having 1.00 millimeters to work with instead of just 1.
I would store a single value in whichever measurement system you choose and so make the calculation only in a single direction each time. So if you choose to store all your values as micrometers as bill30 suggests, then you will always have a consistent value when you convert.
You only need to display whole numbers of millimeters, but you can store at any size you like.
If you only need consistency up to 20 inches then you could just store the full list of conversions in a lookup table, there are only about 500 values if using whole millimeter values.
You only need to display whole numbers of millimeters, but you can store at any size you like.
If you only need consistency up to 20 inches then you could just store the full list of conversions in a lookup table, there are only about 500 values if using whole millimeter values.
ASKER
To maintain the existing capabilities, I am unable to change how the inches are stored. And I would prefer a calculation as it should be faster then a lookup. And then I would need the conversion class to have a database connection.
And I still have to devise a method to generate the stored values. And do I have to have two lookups for converting to and from millimeters, since there are more 32nds then mm?
And I still have to devise a method to generate the stored values. And do I have to have two lookups for converting to and from millimeters, since there are more 32nds then mm?
Is this real data? Are you measuring little parts and keeping track of the inventory?
If so, can you store both numbers?
If you care about accuracy, you really can't use this system.
If you measure in mm and convert to 32nds, 40% of the measurements will be ambiguous.
If so, can you store both numbers?
If you care about accuracy, you really can't use this system.
If you measure in mm and convert to 32nds, 40% of the measurements will be ambiguous.
Exact Exact Rounded
mm inches 32nd
1 0.0313 1
2 0.0625 2 |
3 0.0938 2 |
4 0.1250 3
5 0.1563 4
6 0.1875 5
7 0.2188 6 |
8 0.2500 6 |
9 0.2813 7
10 0.3125 8
11 0.3438 9
12 0.3750 10 |
13 0.4063 10 |
14 0.4375 11
15 0.4688 12
16 0.5000 13 |
17 0.5313 13 |
18 0.5625 14
19 0.5938 15
20 0.6250 16
1. A table lookup of this kind will not be slow at all.
2. You say you are fine with a calculation, you can add the x100 calculation limited to this project, and enter micrometers instead of millimeters as stated before.
3. If none of the answers above suit you just try changing the database value to a decimal instead of an integer, giving you enough digits to be accurate.
2. You say you are fine with a calculation, you can add the x100 calculation limited to this project, and enter micrometers instead of millimeters as stated before.
3. If none of the answers above suit you just try changing the database value to a decimal instead of an integer, giving you enough digits to be accurate.
Since the right side of your mapping is smaller than the left (32 32nds in an inch, but 25mm) it is impossible to map correctly back and forth. Since you cannot change how the inches are stored and since the inches are the more precise, then stop converting back and forth (always a bad idea, try it with Google translator!)
I agree with RobinD. Keep the inches stored as they are and every time you need a mm convert from inches. The only other way would be to store the millimeters with one decimal digit and divide by 10 before use each time.
I agree with RobinD. Keep the inches stored as they are and every time you need a mm convert from inches. The only other way would be to store the millimeters with one decimal digit and divide by 10 before use each time.
But, if you insist, then this is the best option (by far, in my opinion)
Convert to 16ths of an inch on the way back then store it as 32nds. So basically just round to the nearest even integer. This way you have a simple, easy, consistent solution.
Since you know the precision will be off by at least that much anyway, now it's standardized across the whole thing and you don't need to worry about finding patterns (your patterns aren't going to repeat at a useful period, you'll need more rounding and have potential for error in higher numbers)
Rounding to nearest even number of 32nds is robust and easy and as accurate as any other method so I would highly recommend doing it that way.
Convert to 16ths of an inch on the way back then store it as 32nds. So basically just round to the nearest even integer. This way you have a simple, easy, consistent solution.
Since you know the precision will be off by at least that much anyway, now it's standardized across the whole thing and you don't need to worry about finding patterns (your patterns aren't going to repeat at a useful period, you'll need more rounding and have potential for error in higher numbers)
Rounding to nearest even number of 32nds is robust and easy and as accurate as any other method so I would highly recommend doing it that way.
ASKER
I cannot change the stored value without effecting the existing system. And I need to support constraints coded against the stored values.
As long as the resulting converted value is not greater, and only the whole inches are of concern, then using a conversion of 32/25.4 and a fudge factor of .02 works for all but 8 instances in the first 42 feet of measurements. If I use -.02 for those they work also. Doing that means that any mm value can be stored and the resulting value matches the original mm. There is a loss when converting a inches value is about 20% of the cases where the stored value is 1 less or 1/32th of an inch. But I think I can live with that as 32 can't go into 25.4 without some loss.
I was hoping someone would figure out a sign wave or ordinal pattern that would allow me to compensate for the error in a consistent way and not use a hard coded list to avoid conflicts that effect the system. After seeing the graph of the rounding error, it just seemed like something was possible.
As long as the resulting converted value is not greater, and only the whole inches are of concern, then using a conversion of 32/25.4 and a fudge factor of .02 works for all but 8 instances in the first 42 feet of measurements. If I use -.02 for those they work also. Doing that means that any mm value can be stored and the resulting value matches the original mm. There is a loss when converting a inches value is about 20% of the cases where the stored value is 1 less or 1/32th of an inch. But I think I can live with that as 32 can't go into 25.4 without some loss.
I was hoping someone would figure out a sign wave or ordinal pattern that would allow me to compensate for the error in a consistent way and not use a hard coded list to avoid conflicts that effect the system. After seeing the graph of the rounding error, it just seemed like something was possible.
I was hoping someone would figure out a sign wave or ordinal pattern that
It's not a sine wave, it's a linear function. Each time you convert 1/32 you have an extra .7656ish that rolls over. If it was .6 for example, you'd see this pattern in the decimal.
.6, .2, .8, .4, 0, .6 repeating. Problem is that you still lose data because some mm numbers need to go to different inch values. Another problem is that your pattern wouldn't have the same period every time. Sometimes it would 'wrap' in 24 mm, sometimes n 25.
32 can't go into 25.4 without some loss.
Exactly. The right way to do it is to absorb the loss in a fair way. See, no matter what value you get back for the inches, you know it's +/- 1/32 of what it is supposed to be. Remember significant figures from Science or Math classes in High School? Same idea here.
No method you find will give you any better precision than rounding to the nearest 16th of an inch when converting back without changing the structure so you can 'remember' what it came from. Rounding to the nearest 16th will give consistent results (positive and negative error will be equal). It also won't freak out any mathematicians who see your work later.
It's not a sine wave, it's a linear function. Each time you convert 1/32 you have an extra .7656ish that rolls over. If it was .6 for example, you'd see this pattern in the decimal.
.6, .2, .8, .4, 0, .6 repeating. Problem is that you still lose data because some mm numbers need to go to different inch values. Another problem is that your pattern wouldn't have the same period every time. Sometimes it would 'wrap' in 24 mm, sometimes n 25.
32 can't go into 25.4 without some loss.
Exactly. The right way to do it is to absorb the loss in a fair way. See, no matter what value you get back for the inches, you know it's +/- 1/32 of what it is supposed to be. Remember significant figures from Science or Math classes in High School? Same idea here.
No method you find will give you any better precision than rounding to the nearest 16th of an inch when converting back without changing the structure so you can 'remember' what it came from. Rounding to the nearest 16th will give consistent results (positive and negative error will be equal). It also won't freak out any mathematicians who see your work later.
I tried something similar to remove the tax from prices to get the original pre-tax price. The conclusion was that it cannot be done. The information has been removed from the value when the rounding was done and it cannot be retrieved accurately. For consistent results over a known range of values a look-up table (even one stored in an array inside your module and not in a database) will give the best result.
GNiessen you keep saying you dont want to affect the entire system. If you can add an equation to part of the system, you can change the milli to micro for part of the system, or use a table lookup for part of the system.
But if you choose to round some numbers to the closest 32nd and some to the closest inch or half inch, then it's bad math and you'll have bias in your error. When you 'fudge' the ones close to whole inches to whole inches what you are really doing is selectively rounding some numbers to the nearest 16th. If you want some rounded more than to the nearest 16th, then you should do it on all of them.
ASKER
I am not trying to map 32 into 25.4 as much as map the 25.4 into the 32 in a consistent way. If I round to the nearest 16th, then I loose some of the 25.4. I need to be able to support any mm measurement. And match mm to inches with some consistency.
Tommy,
I think you may have something in your ".6, .2, .8, .4, 0, .6 repeating". But it is more like there are two overlapping patterns. If I can model them, then I can map them to a fudge factor that gives me the results i need.
Tommy,
I think you may have something in your ".6, .2, .8, .4, 0, .6 repeating". But it is more like there are two overlapping patterns. If I can model them, then I can map them to a fudge factor that gives me the results i need.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Not a plug and play solution, but I should be able to make it work. Thanks.