Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

rounding error in Powerbuilder

Posted on 2014-02-05
2
Medium Priority
?
535 Views
Last Modified: 2014-02-05
Hello

I want to do this calculation in Powerbuilder Ver 12.1 Build 6639 on Windows 7 64 Bit.

double ld_val_1 = 111.3
double ld_val_2 = 111

double ld_result

ld_result = ld_val_1 - ld_val_2

ld_result should have the value 0.3, but I get the value 0.2999999999999972.

It this a common error or is only my system sick?

Sincerely

Ivo Raemy
0
Comment
Question by:ivoraemy
2 Comments
 
LVL 84

Accepted Solution

by:
David Johnson, CD, MVP earned 1500 total points
ID: 39837862
no it is normal round off the extra numbers and you will get what you desire floating point numbers are never exact when you do calculations on them.  That or use integer math after normalizing i.e.
long val1 =1100
long val2 =1130
long difference
double result = double (difference/100)
0
 

Author Comment

by:ivoraemy
ID: 39838053
Dear Sir

Thank you for your comment.

If I work with the data type decimal (28 digits), it works. Double has 15 digits of precision

decimal ld_val_1 = 111.3
decimal ld_val_2 = 111

decimal ld_result

ld_result = ld_val_1 - ld_val_2

Sincerely

Ivo Raemy
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In real business world data are crucial and sometimes data are shared among different information systems. Hence, an agreeable file transfer protocol need to be established.
If you are a mobile app developer and especially develop hybrid mobile apps then these 4 mistakes you must avoid for hybrid app development to be the more genuine app developer.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
Loops Section Overview
Suggested Courses

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question