C Math functions in RPGIV and floating point variables

This is how I coded from examples on the Internet.  Everything using floating point variables.

Data:

Latitude 1 (LATCPZ): 41.320612

Longitude 1 (LNGCPZ): -81.363520

Latitude 2 (LATCSE): 39.6263358

Longitude 2 (LNGCSE): -84.1956464

Code Example:

D ACos            PR             8F   ExtProc('acos')    
D  ACRadians                     8F   Const              
D Cos             PR             8F   ExtProc('cos')    
D  CRadians                      8F   Const              
D Sin             PR             8F   ExtProc('sin')    
D  SRadians                      8F   Const  

D RLat1           S              8f
D RLat2           S              8f
D RLng1           S              8f
D RLng2           S              8f

D a               S              8f  
D b               S              8f  
D c               S              8f  

c                   Eval      RLat1 = LatCpZ/57.2958                  
c                   Eval      RLng1 = LngCpZ/57.2958                  
c                   Eval      RLat2 = LatCSe/57.2958                  
c                   Eval      RLng2 = LngCSe/57.2958                  
                                                                       
* Calculate distance between locations and convert to miles          
c                   Eval      a = ACos((Sin(RLat1)*Sin(RLat2)) +      
c                             (Cos(RLat1)*Cos(RLat2)*Cos(RLng2-RLng1)))
c                   Eval      b = a * 6371.01                          
c                   Eval      c = b * .62137                            

Expected Result:            188.9 miles

Actual Result:                   3.1067 miles  

Well, without testing (and assuming you have the formula right, and it looks good at a quick glance), there are two likely candidates.

First of all, floating point numeric values are compromises that allow us to represent approximations of numbers in a relatively compact form.  Floating points aren't great when you need a lot of precision, since less-significant digits get lopped off. You're using double-precision floating points here, which give you the ability to approximate some very large and small numbers, but won't give you the precision of large decimal formats for numbers in the restricted ranges that you need here.

Second, the implementation that you have chosen is apparently sensitive to rounding errors, especially at small distances.  You might want to consider trying the Haversine formula, or the somewhat more complex special case of the Vincenty formula:

http://en.wikipedia.org/wiki/Great-circle_distance

- Gary Patterson

Check out my EE profile:  http://www.experts-exchange.com/M_4382324.html

BTW, what are the definitions of the CpZ and CSe variables? Also, I don't think I'd break the formula into three separate a, b and c components. Each one separately probably introduces additional conversion/precision errors. Use a single unified formula. And that probably could be applied to the constants too -- provide them with declared data types, most likely 8F for each of them.

Tom

thanks.  I'll give both suggestions a try.