• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 541
  • Last Modified:

rounding error in Powerbuilder


I want to do this calculation in Powerbuilder Ver 12.1 Build 6639 on Windows 7 64 Bit.

double ld_val_1 = 111.3
double ld_val_2 = 111

double ld_result

ld_result = ld_val_1 - ld_val_2

ld_result should have the value 0.3, but I get the value 0.2999999999999972.

It this a common error or is only my system sick?


Ivo Raemy
1 Solution
David Johnson, CD, MVPOwnerCommented:
no it is normal round off the extra numbers and you will get what you desire floating point numbers are never exact when you do calculations on them.  That or use integer math after normalizing i.e.
long val1 =1100
long val2 =1130
long difference
double result = double (difference/100)
ivoraemyAuthor Commented:
Dear Sir

Thank you for your comment.

If I work with the data type decimal (28 digits), it works. Double has 15 digits of precision

decimal ld_val_1 = 111.3
decimal ld_val_2 = 111

decimal ld_result

ld_result = ld_val_1 - ld_val_2


Ivo Raemy
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now