GetItemNumber

In PB 8.0.4 I have a datawindow and mycolumn column with the following values:
19.5
20.0
19.5
19.5
19.5
20.0
20.0
19.5
19.5
19.5

If round the average to 1 decimal place:

The formula for compute_4 in the datawindow is
compute_4 = round(avg(mycolumn for all),1) = 19.7

In a command button script I have
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is 19.70000000000000000


If round to 2 decimal places
round(avg(mycolumn for all),2) and the result 19.65
Ld_c4 = dw_measure_body.getitemnumber( 1, "compute_4") and the value is  19.64999999999999744

Why Ld_c4 is not 19.65 in this last case?

Thank you.

magoticiAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

diasroshanCommented:
hi,
if u want 1 decimal place do the following...


//Declare decimal variable as
Decimal{1} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")


if u want 2 decimal place do the following...


//Declare decimal variable as
Decimal{2} ld_c4

// dont use getitemnumber... use getitemdecimal
Ld_c4 = dw_measure_body.getitemdecimal( 1, "compute_4")


Let me know if u need more assistance....


Cheers,
Rosh
0
diasroshanCommented:
hi,

also specify a 'format' for ur computed field as per ur decimal places required...

Cheers,
Rosh
0
berXpertCommented:
Hi,

Check your datatype in Ld_c4, maybe you are using a Double or a Real.

You still can use GetItemNumber but need to change Ld_c4 to Decimal {2}

regards,

BerX
0
Cloud Class® Course: SQL Server Core 2016

This course will introduce you to SQL Server Core 2016, as well as teach you about SSMS, data tools, installation, server configuration, using Management Studio, and writing and executing queries.

magoticiAuthor Commented:
If you take a calculator will see that the average for the values in mycolumn is exactly 19.65 with no other decimals.
19.65 rouned to one decimal place is 19.7
19.64999.... is 19.6
Where is that number of decimals coming from. Why the value that comes from the DW is not exact?
Using decimal{2} will only do a rounding of the value that comes from the DW.
0
falvaro3Commented:
Did you modify the Round datawindow expression to return 2 decimal places?
I see in your original posting, you have "round(avg(mycolumn for all),1)".  If you change that to " round(avg(mycolumn for all),2)", does that resolve your issue.  It seems as though the rounding will occur on only the next decimal place specified, i.e.: if round decimal precision is set to 1, 19.65 uses the "5" for rounding.  For 19.64999, it'll use ONLY the "4".  If you set the precision to 2, you should get the rounding on the "9" in the third decimal place.
Just a guess; havent tested (yet).
0
magoticiAuthor Commented:
round(avg(mycolumn for all),1) will display into the DW 19.7
round(avg(mycolumn for all),2) will display 19.65 but getitemdecimal or getitemnumber will bring into the script 19.64999999999999744 instead of 19.65
I repeat, with a calculator the result is 19.65 not 19.64999...
0
falvaro3Commented:
What about Rounding (again)?

ld_c4 = Round(dw_measure_body.GetItemDecimal(1, "compute_4"), 2)


(I know, sounds redundant to round twice, but does it work?)  ;)
0
falvaro3Commented:
Also, I'd be sure to set the format on the computed field to something like
    #,##0.00
and also declare the decimal var in script as
     Decimal{4}   ld_c4

(as previously mentioned)
0
magoticiAuthor Commented:
Decimal will do a round(x, 4) , my question is why getitem decimal returns that funny number, is it a bug or  what?
0
falvaro3Commented:
it's a bug.  
or a what.
not sure, can't dupe in pb9.
I set up an external dw with your data, and performed the steps you took, but always got the correct value in ld_c4.
<*boggle*>.  
I'd check to see if there are any new builds/patches you're missing for pb8 (not sure if sybase has them or not).
0
BhattiCommented:
Hi,

GetitemDecimal or GetItemNumber will return you the values as you described. But to show the values you need to develop in your datawindow the column format and so in which format you want to see the value.More over please check also the setting in the system.

Return value you can see in datawindow or some fields as editmask etc. depends upon you setting. Check the values with messagebox("Test", string(value, "hier_format"))

Please let me know you results.

Best regards

Bhatti
0
magoticiAuthor Commented:
For
messagebox("Test", string(value, "0.0")) I've got  9.6
and for
messagebox("Test", string(value, "0.00")) the displayed value is  9.65

The value 19.64999999999999744 is in debug mode.

The table column is a decimal(5,1) in ASA 8 db.

0
BhattiCommented:
Hi,
value (19.65) you are getting is the result of the multiplcation of some values r division or some other mathimatical calculations. Please let me know under which calculation you are getting this value.

Calculater will show you only two digitis after comma.

Do this in Excel:

write the formulla giving you result 19.65
Now with the help of rightmousebutton activate the format. select number and after dot select 10 and see the result. must be the same as you get in powerbuilder.

Please let me know the result.

Best regards

Bhatti
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
magoticiAuthor Commented:
I used avg() in dw computed field and average() in Excel .
The result in Excel is 19.650000...

magotici
0
BhattiCommented:
Hi,

May you are dividing or multiplying values long and double or with float mixing or any other such condition. May this cause you a such result.


Best regards

Bhatti
0
magoticiAuthor Commented:
THE VALUES ARE DECIMAL(5,1)

MAGOTICI
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Editors IDEs

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.