Solved

rounding error in Powerbuilder

Posted on 2014-02-05
2
476 Views
Last Modified: 2014-02-05
Hello

I want to do this calculation in Powerbuilder Ver 12.1 Build 6639 on Windows 7 64 Bit.

double ld_val_1 = 111.3
double ld_val_2 = 111

double ld_result

ld_result = ld_val_1 - ld_val_2

ld_result should have the value 0.3, but I get the value 0.2999999999999972.

It this a common error or is only my system sick?

Sincerely

Ivo Raemy
0
Comment
Question by:ivoraemy
2 Comments
 
LVL 79

Accepted Solution

by:
David Johnson, CD, MVP earned 500 total points
ID: 39837862
no it is normal round off the extra numbers and you will get what you desire floating point numbers are never exact when you do calculations on them.  That or use integer math after normalizing i.e.
long val1 =1100
long val2 =1130
long difference
double result = double (difference/100)
0
 

Author Comment

by:ivoraemy
ID: 39838053
Dear Sir

Thank you for your comment.

If I work with the data type decimal (28 digits), it works. Double has 15 digits of precision

decimal ld_val_1 = 111.3
decimal ld_val_2 = 111

decimal ld_result

ld_result = ld_val_1 - ld_val_2

Sincerely

Ivo Raemy
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you upgrade from Windows 8 to 8.1 or to Windows 10 or if you are like me you are on the Insider Program you may find yourself with many 450MB recovery partitions.  With a traditional disk that may not be a problem but with relatively smaller SS…
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.

816 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now