magotici
asked on
GetItemNumber
In PB 8.0.4 I have a datawindow and mycolumn column with the following values:
19.5
20.0
19.5
19.5
19.5
20.0
20.0
19.5
19.5
19.5
If round the average to 1 decimal place:
The formula for compute_4 in the datawindow is
compute_4 = round(avg(mycolumn for all),1) = 19.7
In a command button script I have
Ld_c4 = dw_measure_body.getitemnum
If round to 2 decimal places
round(avg(mycolumn for all),2) and the result 19.65
Ld_c4 = dw_measure_body.getitemnum
Why Ld_c4 is not 19.65 in this last case?
Thank you.
19.5
20.0
19.5
19.5
19.5
20.0
20.0
19.5
19.5
19.5
If round the average to 1 decimal place:
The formula for compute_4 in the datawindow is
compute_4 = round(avg(mycolumn for all),1) = 19.7
In a command button script I have
Ld_c4 = dw_measure_body.getitemnum
If round to 2 decimal places
round(avg(mycolumn for all),2) and the result 19.65
Ld_c4 = dw_measure_body.getitemnum
Why Ld_c4 is not 19.65 in this last case?
Thank you.
hi,
also specify a 'format' for ur computed field as per ur decimal places required...
Cheers,
Rosh
also specify a 'format' for ur computed field as per ur decimal places required...
Cheers,
Rosh
Hi,
Check your datatype in Ld_c4, maybe you are using a Double or a Real.
You still can use GetItemNumber but need to change Ld_c4 to Decimal {2}
regards,
BerX
Check your datatype in Ld_c4, maybe you are using a Double or a Real.
You still can use GetItemNumber but need to change Ld_c4 to Decimal {2}
regards,
BerX
ASKER
If you take a calculator will see that the average for the values in mycolumn is exactly 19.65 with no other decimals.
19.65 rouned to one decimal place is 19.7
19.64999.... is 19.6
Where is that number of decimals coming from. Why the value that comes from the DW is not exact?
Using decimal{2} will only do a rounding of the value that comes from the DW.
19.65 rouned to one decimal place is 19.7
19.64999.... is 19.6
Where is that number of decimals coming from. Why the value that comes from the DW is not exact?
Using decimal{2} will only do a rounding of the value that comes from the DW.
Did you modify the Round datawindow expression to return 2 decimal places?
I see in your original posting, you have "round(avg(mycolumn for all),1)". If you change that to " round(avg(mycolumn for all),2)", does that resolve your issue. It seems as though the rounding will occur on only the next decimal place specified, i.e.: if round decimal precision is set to 1, 19.65 uses the "5" for rounding. For 19.64999, it'll use ONLY the "4". If you set the precision to 2, you should get the rounding on the "9" in the third decimal place.
Just a guess; havent tested (yet).
I see in your original posting, you have "round(avg(mycolumn for all),1)". If you change that to " round(avg(mycolumn for all),2)", does that resolve your issue. It seems as though the rounding will occur on only the next decimal place specified, i.e.: if round decimal precision is set to 1, 19.65 uses the "5" for rounding. For 19.64999, it'll use ONLY the "4". If you set the precision to 2, you should get the rounding on the "9" in the third decimal place.
Just a guess; havent tested (yet).
ASKER
round(avg(mycolumn for all),1) will display into the DW 19.7
round(avg(mycolumn for all),2) will display 19.65 but getitemdecimal or getitemnumber will bring into the script 19.64999999999999744 instead of 19.65
I repeat, with a calculator the result is 19.65 not 19.64999...
round(avg(mycolumn for all),2) will display 19.65 but getitemdecimal or getitemnumber will bring into the script 19.64999999999999744 instead of 19.65
I repeat, with a calculator the result is 19.65 not 19.64999...
What about Rounding (again)?
ld_c4 = Round(dw_measure_body.GetI
(I know, sounds redundant to round twice, but does it work?) ;)
ld_c4 = Round(dw_measure_body.GetI
(I know, sounds redundant to round twice, but does it work?) ;)
Also, I'd be sure to set the format on the computed field to something like
#,##0.00
and also declare the decimal var in script as
Decimal{4} ld_c4
(as previously mentioned)
#,##0.00
and also declare the decimal var in script as
Decimal{4} ld_c4
(as previously mentioned)
ASKER
Decimal will do a round(x, 4) , my question is why getitem decimal returns that funny number, is it a bug or what?
it's a bug.
or a what.
not sure, can't dupe in pb9.
I set up an external dw with your data, and performed the steps you took, but always got the correct value in ld_c4.
<*boggle*>.
I'd check to see if there are any new builds/patches you're missing for pb8 (not sure if sybase has them or not).
or a what.
not sure, can't dupe in pb9.
I set up an external dw with your data, and performed the steps you took, but always got the correct value in ld_c4.
<*boggle*>.
I'd check to see if there are any new builds/patches you're missing for pb8 (not sure if sybase has them or not).
Hi,
GetitemDecimal or GetItemNumber will return you the values as you described. But to show the values you need to develop in your datawindow the column format and so in which format you want to see the value.More over please check also the setting in the system.
Return value you can see in datawindow or some fields as editmask etc. depends upon you setting. Check the values with messagebox("Test", string(value, "hier_format"))
Please let me know you results.
Best regards
Bhatti
GetitemDecimal or GetItemNumber will return you the values as you described. But to show the values you need to develop in your datawindow the column format and so in which format you want to see the value.More over please check also the setting in the system.
Return value you can see in datawindow or some fields as editmask etc. depends upon you setting. Check the values with messagebox("Test", string(value, "hier_format"))
Please let me know you results.
Best regards
Bhatti
ASKER
For
messagebox("Test", string(value, "0.0")) I've got 9.6
and for
messagebox("Test", string(value, "0.00")) the displayed value is 9.65
The value 19.64999999999999744 is in debug mode.
The table column is a decimal(5,1) in ASA 8 db.
messagebox("Test", string(value, "0.0")) I've got 9.6
and for
messagebox("Test", string(value, "0.00")) the displayed value is 9.65
The value 19.64999999999999744 is in debug mode.
The table column is a decimal(5,1) in ASA 8 db.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I used avg() in dw computed field and average() in Excel .
The result in Excel is 19.650000...
magotici
The result in Excel is 19.650000...
magotici
Hi,
May you are dividing or multiplying values long and double or with float mixing or any other such condition. May this cause you a such result.
Best regards
Bhatti
May you are dividing or multiplying values long and double or with float mixing or any other such condition. May this cause you a such result.
Best regards
Bhatti
ASKER
THE VALUES ARE DECIMAL(5,1)
MAGOTICI
MAGOTICI
if u want 1 decimal place do the following...
//Declare decimal variable as
Decimal{1} ld_c4
// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdec
if u want 2 decimal place do the following...
//Declare decimal variable as
Decimal{2} ld_c4
// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdec
Let me know if u need more assistance....
Cheers,
Rosh