• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1872
  • Last Modified:

GetItemNumber

In PB 8.0.4 I have a datawindow and mycolumn column with the following values:
19.5
20.0
19.5
19.5
19.5
20.0
20.0
19.5
19.5
19.5

If round the average to 1 decimal place:

The formula for compute_4 in the datawindow is
compute_4 = round(avg(mycolumn for all),1) = 19.7

In a command button script I have
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is 19.70000000000000000


If round to 2 decimal places
round(avg(mycolumn for all),2) and the result 19.65
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is  19.64999999999999744

Why Ld_c4 is not 19.65 in this last case?

Thank you.

0
magotici
Asked:
magotici
  • 6
  • 4
  • 3
  • +2
1 Solution
 
diasroshanCommented:
hi,
if u want 1 decimal place do the following...


//Declare decimal variable as
Decimal{1} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")


if u want 2 decimal place do the following...


//Declare decimal variable as
Decimal{2} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")


Let me know if u need more assistance....


Cheers,
Rosh
0
 
diasroshanCommented:
hi,

also specify a 'format' for ur computed field as per ur decimal places required...

Cheers,
Rosh
0
 
berXpertCommented:
Hi,

Check your datatype in Ld_c4, maybe you are using a Double or a Real.

You still can use GetItemNumber but need to change Ld_c4 to Decimal {2}

regards,

BerX
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
magoticiAuthor Commented:
If you take a calculator will see that the average for the values in mycolumn is exactly 19.65 with no other decimals.
19.65 rouned to one decimal place is 19.7
19.64999.... is 19.6
Where is that number of decimals coming from. Why the value that comes from the DW is not exact?
Using decimal{2} will only do a rounding of the value that comes from the DW.
0
 
falvaro3Commented:
Did you modify the Round datawindow expression to return 2 decimal places?
I see in your original posting, you have "round(avg(mycolumn for all),1)".  If you change that to " round(avg(mycolumn for all),2)", does that resolve your issue.  It seems as though the rounding will occur on only the next decimal place specified, i.e.: if round decimal precision is set to 1, 19.65 uses the "5" for rounding.  For 19.64999, it'll use ONLY the "4".  If you set the precision to 2, you should get the rounding on the "9" in the third decimal place.
Just a guess; havent tested (yet).
0
 
magoticiAuthor Commented:
round(avg(mycolumn for all),1) will display into the DW 19.7
round(avg(mycolumn for all),2) will display 19.65 but getitemdecimal or getitemnumber will bring into the script 19.64999999999999744 instead of 19.65
I repeat, with a calculator the result is 19.65 not 19.64999...
0
 
falvaro3Commented:
What about Rounding (again)?

ld_c4 = Round(dw_measure_body.GetItemDecimal(1, "compute_4"), 2)


(I know, sounds redundant to round twice, but does it work?)  ;)
0
 
falvaro3Commented:
Also, I'd be sure to set the format on the computed field to something like
    #,##0.00
and also declare the decimal var in script as
     Decimal{4}   ld_c4

(as previously mentioned)
0
 
magoticiAuthor Commented:
Decimal will do a round(x, 4) , my question is why getitem decimal returns that funny number, is it a bug or  what?
0
 
falvaro3Commented:
it's a bug.  
or a what.
not sure, can't dupe in pb9.
I set up an external dw with your data, and performed the steps you took, but always got the correct value in ld_c4.
<*boggle*>.  
I'd check to see if there are any new builds/patches you're missing for pb8 (not sure if sybase has them or not).
0
 
BhattiCommented:
Hi,

GetitemDecimal or GetItemNumber will return you the values as you described. But to show the values you need to develop in your datawindow the column format and so in which format you want to see the value.More over please check also the setting in the system.

Return value you can see in datawindow or some fields as editmask etc. depends upon you setting. Check the values with messagebox("Test", string(value, "hier_format"))

Please let me know you results.

Best regards

Bhatti
0
 
magoticiAuthor Commented:
For
messagebox("Test", string(value, "0.0")) I've got  9.6
and for
messagebox("Test", string(value, "0.00")) the displayed value is  9.65

The value 19.64999999999999744 is in debug mode.

The table column is a decimal(5,1) in ASA 8 db.

0
 
BhattiCommented:
Hi,
value (19.65) you are getting is the result of the multiplcation of some values r division or some other mathimatical calculations. Please let me know under which calculation you are getting this value.

Calculater will show you only two digitis after comma.

Do this in Excel:

write the formulla giving you result 19.65
Now with the help of rightmousebutton activate the format. select number and after dot select 10 and see the result. must be the same as you get in powerbuilder.

Please let me know the result.

Best regards

Bhatti
0
 
magoticiAuthor Commented:
I used avg() in dw computed field and average() in Excel .
The result in Excel is 19.650000...

magotici
0
 
BhattiCommented:
Hi,

May you are dividing or multiplying values long and double or with float mixing or any other such condition. May this cause you a such result.


Best regards

Bhatti
0
 
magoticiAuthor Commented:
THE VALUES ARE DECIMAL(5,1)

MAGOTICI
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 6
  • 4
  • 3
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now