Solved

# GetItemNumber

Posted on 2004-11-17
1,658 Views
In PB 8.0.4 I have a datawindow and mycolumn column with the following values:
19.5
20.0
19.5
19.5
19.5
20.0
20.0
19.5
19.5
19.5

If round the average to 1 decimal place:

The formula for compute_4 in the datawindow is
compute_4 = round(avg(mycolumn for all),1) = 19.7

In a command button script I have
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is 19.70000000000000000

If round to 2 decimal places
round(avg(mycolumn for all),2) and the result 19.65
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is  19.64999999999999744

Why Ld_c4 is not 19.65 in this last case?

Thank you.

0
Question by:magotici

LVL 18

Expert Comment

hi,
if u want 1 decimal place do the following...

//Declare decimal variable as
Decimal{1} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")

if u want 2 decimal place do the following...

//Declare decimal variable as
Decimal{2} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")

Let me know if u need more assistance....

Cheers,
Rosh
0

LVL 18

Expert Comment

hi,

also specify a 'format' for ur computed field as per ur decimal places required...

Cheers,
Rosh
0

LVL 5

Expert Comment

Hi,

Check your datatype in Ld_c4, maybe you are using a Double or a Real.

You still can use GetItemNumber but need to change Ld_c4 to Decimal {2}

regards,

BerX
0

Author Comment

If you take a calculator will see that the average for the values in mycolumn is exactly 19.65 with no other decimals.
19.65 rouned to one decimal place is 19.7
19.64999.... is 19.6
Where is that number of decimals coming from. Why the value that comes from the DW is not exact?
Using decimal{2} will only do a rounding of the value that comes from the DW.
0

LVL 4

Expert Comment

Did you modify the Round datawindow expression to return 2 decimal places?
I see in your original posting, you have "round(avg(mycolumn for all),1)".  If you change that to " round(avg(mycolumn for all),2)", does that resolve your issue.  It seems as though the rounding will occur on only the next decimal place specified, i.e.: if round decimal precision is set to 1, 19.65 uses the "5" for rounding.  For 19.64999, it'll use ONLY the "4".  If you set the precision to 2, you should get the rounding on the "9" in the third decimal place.
Just a guess; havent tested (yet).
0

Author Comment

round(avg(mycolumn for all),1) will display into the DW 19.7
round(avg(mycolumn for all),2) will display 19.65 but getitemdecimal or getitemnumber will bring into the script 19.64999999999999744 instead of 19.65
I repeat, with a calculator the result is 19.65 not 19.64999...
0

LVL 4

Expert Comment

ld_c4 = Round(dw_measure_body.GetItemDecimal(1, "compute_4"), 2)

(I know, sounds redundant to round twice, but does it work?)  ;)
0

LVL 4

Expert Comment

Also, I'd be sure to set the format on the computed field to something like
#,##0.00
and also declare the decimal var in script as
Decimal{4}   ld_c4

(as previously mentioned)
0

Author Comment

Decimal will do a round(x, 4) , my question is why getitem decimal returns that funny number, is it a bug or  what?
0

LVL 4

Expert Comment

it's a bug.
or a what.
not sure, can't dupe in pb9.
I set up an external dw with your data, and performed the steps you took, but always got the correct value in ld_c4.
<*boggle*>.
I'd check to see if there are any new builds/patches you're missing for pb8 (not sure if sybase has them or not).
0

LVL 4

Expert Comment

Hi,

GetitemDecimal or GetItemNumber will return you the values as you described. But to show the values you need to develop in your datawindow the column format and so in which format you want to see the value.More over please check also the setting in the system.

Return value you can see in datawindow or some fields as editmask etc. depends upon you setting. Check the values with messagebox("Test", string(value, "hier_format"))

Please let me know you results.

Best regards

Bhatti
0

Author Comment

For
messagebox("Test", string(value, "0.0")) I've got  9.6
and for
messagebox("Test", string(value, "0.00")) the displayed value is  9.65

The value 19.64999999999999744 is in debug mode.

The table column is a decimal(5,1) in ASA 8 db.

0

LVL 4

Accepted Solution

Hi,
value (19.65) you are getting is the result of the multiplcation of some values r division or some other mathimatical calculations. Please let me know under which calculation you are getting this value.

Calculater will show you only two digitis after comma.

Do this in Excel:

write the formulla giving you result 19.65
Now with the help of rightmousebutton activate the format. select number and after dot select 10 and see the result. must be the same as you get in powerbuilder.

Please let me know the result.

Best regards

Bhatti
0

Author Comment

I used avg() in dw computed field and average() in Excel .
The result in Excel is 19.650000...

magotici
0

LVL 4

Expert Comment

Hi,

May you are dividing or multiplying values long and double or with float mixing or any other such condition. May this cause you a such result.

Best regards

Bhatti
0

Author Comment

THE VALUES ARE DECIMAL(5,1)

MAGOTICI
0

## Featured Post

In our object-oriented world the class is a minimal unit, a brick for constructing our applications. It is an abstraction and we know well how to use it. In well-designed software we are not usually interested in knowing how objects look in memory. …
How to install Selenium IDE and loops for quick automated testing. Get Selenium IDE from http://seleniumhq.org (http://seleniumhq.org) Go to that link and select download selenium in the right hand columnThat will then direct you to their downlo…
The viewer will learn how to use NetBeans IDE 8.0 for Windows to connect to a MySQL database. Open Services Panel: Create a new connection using New Connection Wizard: Create a test database called eetutorial: Create a new test tabel called ee…
The viewer will learn how to use and create keystrokes in Netbeans IDE 8.0 for Windows.