Link to home
Start Free TrialLog in
Avatar of Theo Kouwenhoven
Theo KouwenhovenFlag for Netherlands

asked on

Error in RPG calculation

Hi Experts,

Not a real question but a remark / warning
I discovered an error in a RPG calculation... I use this for years, I don't know How often this happend,
and howmany unhappy customers I have :)

I discovered that the following calculation gives an incorrect result:

Result = 60 / (10**2) = 0,599999
Result = 60 / (10**2) = 0,0599999

While:
Result = 60 / (10*10) = 0,60
Result = 60 / %int(10**2) = 0,60

Conclusion %int(10**2) <> 10**2

This goes wrong for all figures ending with???
I did soms tests and first it look like that every multiple of 3 is wrong,
3, 6, 9, 12, 15, 18, 21, 24

But 27 is Oke and after that the range continues with:

29, 30, 31, 35, 36, 37, 41, 42, 43

For total range see attached file

Column NORM = #Nbr / (10**2)
Column INT  = #Nbr / %int(10**2)

Open in new window


(I can't post this with 0 points)

Regards,
Murph
1to100.txt
Avatar of Member_2_276102
Member_2_276102

I assume there's a typo in these two lines:
Result = 60 / (10**2) = 0,599999
Result = 60 / (10**2) = 0,0599999

In any case, this is the way things have always been. One way you might control it like this:
eval(H)   Result = 60 / (10**2)

Open in new window

The result should be 0,600000.

The "half-adjust" operation extender exists for you to decide how rounding in internal calculations ought to be handled. There are other possibilities that depend on your data types and sizes. The explicit use of integers with %INT() is one additional way.

Tom
Avatar of Theo Kouwenhoven

ASKER

Tom,

A divide by 100 for a result field that is large enough, never requires "half-adjust"
Half-adjust should be a tool for rounding, not for bug-fixing :-)

In this case 60 was a numeric 5,0 and the result field 15,5
I will run the same test again (tomorrow) with
- both fields 15,5
- a packed version of this.
- negative numbers.

So I still wondering why 10^2 is not the same as 100

I will test also the version:
A = 10^2  (result 15,9)
B = 60 / A

I'm quite sure that this result will be 0,60.

I'm wondering if this is/was the same within all OS versions?

In this case 60 was a numeric 5,0 and the result field 15,5

And what was the data type and size of "/ (10**2)"?

The compiler won't perform operations during the compile in order to determine "hmmm... (10**2) is just 100, so I can simply move the decimal point; I don't need to create any intermediate fields and convert it all to binary values and run calculations." It can short-circuit various sequences, but humans can be better at it.

Try it without the exponentiation operator. I.e., just use (100) if that's all you really need.

Tom
Hi Tom,
 
I tested with other options:
 
- values 1 to 100 with a Signd 15,9 result field is gives the same result as packed 15,9.
 
- the same problem for -100 to -1
 
- data type and size of "/ (10**2)".... Yes that is the problem it's part of a "complex" calculation
 
- In two steps give the right result
   A = 10^2  (result 15,9)
   B = 60 / A
 

Its part of a routine change an alpha field to a numeric resuly field with a variable number of decimals.
Rerult = #Str2Dec(CharValue:DecPos)
 
So the value of decimals can be 1**0 to 1**9.
of cause I can solve it with Select; when DecPos = 0; etc.
But this is a more elegant way (I think).


Its solved by using the %INT, but I'm sure I'm not the first one who discovered this?
So still wondering why it goes wrong, and if it is a bug in all releases....
Can I find if it is already reported by IBM?


Not much I can add. The oldest easily available ILE RPG reference I have is for V5R1. In chapter 21, Expressions : Data Types : Format of Numeric Intermediate Results : For the ** operator:
The result is represented in float format.
As far as I know, it's always been that way (i.e., as long as RPG has had an exponentiation operator.) Any conversion of data type between float, integer and/or packed (or zoned) is always subject to rounding error.

By itself, that's not too troublesome. But just before that, under For the / operator:, there is also this:
If one operand is float or the FLTDIV keyword is specified on the control
specification, then the result of the / operator is float. Otherwise the result is
packed-decimal.
So, the exponentiation gets a float involved, and the division subsequently does its work giving a float result. When that result is turned into a packed (or zoned) final result, a tiny bit gets lost out at the far right. If you don't ask for adjustment, you lose it.

Tom
Hello Dhaest,

It was not a question, just reporting a bug and a discussion possibility,
Do whatever you want, as long as this stay as a searchable topic op EE

ASKER CERTIFIED SOLUTION
Avatar of Member_2_276102
Member_2_276102

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial