Solved

# rounding error in Powerbuilder

Posted on 2014-02-05
467 Views
Hello

I want to do this calculation in Powerbuilder Ver 12.1 Build 6639 on Windows 7 64 Bit.

double ld_val_1 = 111.3
double ld_val_2 = 111

double ld_result

ld_result = ld_val_1 - ld_val_2

ld_result should have the value 0.3, but I get the value 0.2999999999999972.

It this a common error or is only my system sick?

Sincerely

Ivo Raemy
0
Question by:ivoraemy

LVL 78

Accepted Solution

David Johnson, CD, MVP earned 500 total points
ID: 39837862
no it is normal round off the extra numbers and you will get what you desire floating point numbers are never exact when you do calculations on them.  That or use integer math after normalizing i.e.
long val1 =1100
long val2 =1130
long difference
double result = double (difference/100)
0

Author Comment

ID: 39838053
Dear Sir

If I work with the data type decimal (28 digits), it works. Double has 15 digits of precision

decimal ld_val_1 = 111.3
decimal ld_val_2 = 111

decimal ld_result

ld_result = ld_val_1 - ld_val_2

Sincerely

Ivo Raemy
0

## Featured Post

### Suggested Solutions

Fibonacci challenge 11 84
How does Google get such fast response times? 5 126
creating threads in delphi 1 55
This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
This is about my first experience with programming Arduino.
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…