Solved

XPath and XQuery, formating and precision

Posted on 2015-01-22
3
79 Views
Last Modified: 2015-01-22
From my fn:sum command, I get the following:

1.0001865499999999E6

Open in new window


where I should be getting

1000186.55

Open in new window


How do I fix that
0
Comment
Question by:Anthony Lucia
  • 2
3 Comments
 

Author Comment

by:Anthony Lucia
ID: 40565336
I do understand that perhaps a format-number() function might help, but what would cause the lack of precision
0
 
LVL 60

Accepted Solution

by:
Geert Bormans earned 500 total points
ID: 40565362
format-number() would fix that, but not entirely

numbers in XPath are of type xs:double
xs:double has floating point... so they are not decimals
avoid the issue by explicitely casting to xs:decimal prior to the sum()

instead of sum(node) do sum(node/xs:decimal())

if you are bound to XPath 1.0 it might be a lot trickier
0
 
LVL 60

Assisted Solution

by:Geert Bormans
Geert Bormans earned 500 total points
ID: 40565370
http://en.wikipedia.org/wiki/Floating_point
has a section on accuracy issues
decimals don't have that
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Many times as a report developer I've been asked to display normalized data such as three rows with values Jack, Joe, and Bob as a single comma-separated string such as 'Jack, Joe, Bob', and vice versa.  Here's how to do it. 
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now