Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

rounding error in Powerbuilder

Posted on 2014-02-05
2
Medium Priority
?
522 Views
Last Modified: 2014-02-05
Hello

I want to do this calculation in Powerbuilder Ver 12.1 Build 6639 on Windows 7 64 Bit.

double ld_val_1 = 111.3
double ld_val_2 = 111

double ld_result

ld_result = ld_val_1 - ld_val_2

ld_result should have the value 0.3, but I get the value 0.2999999999999972.

It this a common error or is only my system sick?

Sincerely

Ivo Raemy
0
Comment
Question by:ivoraemy
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 83

Accepted Solution

by:
David Johnson, CD, MVP earned 1500 total points
ID: 39837862
no it is normal round off the extra numbers and you will get what you desire floating point numbers are never exact when you do calculations on them.  That or use integer math after normalizing i.e.
long val1 =1100
long val2 =1130
long difference
double result = double (difference/100)
0
 

Author Comment

by:ivoraemy
ID: 39838053
Dear Sir

Thank you for your comment.

If I work with the data type decimal (28 digits), it works. Double has 15 digits of precision

decimal ld_val_1 = 111.3
decimal ld_val_2 = 111

decimal ld_result

ld_result = ld_val_1 - ld_val_2

Sincerely

Ivo Raemy
0

Featured Post

Build and deliver software with DevOps

A digital transformation requires faster time to market, shorter software development lifecycles, and the ability to adapt rapidly to changing customer demands. DevOps provides the solution.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We live in a world of interfaces like the one in the title picture. VBA also allows to use interfaces which offers a lot of possibilities. This article describes how to use interfaces in VBA and how to work around their bugs.
This article will show how Aten was able to supply easy management and control for Artear's video walls and wide range display configurations of their newsroom.
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.
Windows 8 came with a dramatically different user interface known as Metro. Notably missing from that interface was a Start button and Start Menu. Microsoft responded to negative user feedback of the Metro interface, bringing back the Start button a…

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question