• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 97
  • Last Modified:

XPath and XQuery, formating and precision

From my fn:sum command, I get the following:

1.0001865499999999E6

Open in new window


where I should be getting

1000186.55

Open in new window


How do I fix that
0
Anthony Lucia
Asked:
Anthony Lucia
  • 2
2 Solutions
 
Anthony LuciaAuthor Commented:
I do understand that perhaps a format-number() function might help, but what would cause the lack of precision
0
 
Geert BormansInformation ArchitectCommented:
format-number() would fix that, but not entirely

numbers in XPath are of type xs:double
xs:double has floating point... so they are not decimals
avoid the issue by explicitely casting to xs:decimal prior to the sum()

instead of sum(node) do sum(node/xs:decimal())

if you are bound to XPath 1.0 it might be a lot trickier
0
 
Geert BormansInformation ArchitectCommented:
http://en.wikipedia.org/wiki/Floating_point
has a section on accuracy issues
decimals don't have that
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now