Solved

How to weight multiple estimates

Posted on 2011-03-06
14
360 Views
Last Modified: 2012-08-14
This question is pretty long-winded, so please bear with me...

Two people have estimated a quantity, call it xm, with results A and B. (To put a practical perspective on it, xm could be the cost of a software project or the weight of the fuel for the space shuttle.)
A and B are range estimates (a1, a2), and (b1, b2), with means am, bm, and variances av, bv, respectively.
Assume you know from controlled tests that:
 - A and B are normally distributed.
 - A and B have a correlation of 0.
 - (a1, a2) and (b1, b2) are 90% confidence intervals (ie, there is a 90% chance that a1<=X<=a2, and a 90% chance that b1<=X<=b2).

In order to get the best estimate for xm, you want to use an average of A and B, ie.,
  xm = (1-k)am + kbm
where 0<k<1.

But (and this is my question, finally), what should k be?

NOTE: Intuitively if av=bv, then k should be 0.5, but, also intuitively, if av<bv then k should be >0.5. To illustrate the last point, say (a1, a2) is (250,260) and (b1, b2) is (0,1000). Wouldn't you put more weight on am?


PS. I'm not a student, and this is not a homework question.
0
Comment
Question by:Milewskp
  • 7
  • 3
  • 3
14 Comments
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 35046915
Not at all. If they are both actually 90% confident, then they should be weighted equally. If you were trying to find the average salary in a company and one guy made twice as much as anyone else, you still weight his the same (1/n). Just because some number is bigger doesn't mean the weight should be different.

Actually, because of the way the variance is squared, the higher variance is sort of weighted higher than the other.
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 35046922
Now, there is one very strange point in this problem (which has been mentioned before). You say that two different estimates of the same number are both 90% accurate, but they are incredibly different. This isn't really possible which is why the intuition is wrong. Any time you try to think intuitively about an impossible situation, you get odd conclusions.
0
 
LVL 1

Author Comment

by:Milewskp
ID: 35048261
Hi Tommy,
<If they are both actually 90% confident, then they should be weighted equally>
Consider this example:
Say the first estimate for the amount of fuel for the space shuttle is 35,000 - 36,000 lbs, and the second estimate is 1000 - 100,000 lbs.
Intuitively, if both estimators give answers that are correct 90% of the time, it's a pretty reasonable assumption that the guy with the tighter estimate is more knowledgable than the other guy (he's probably a NASA engineer to be 90% confident of such a tight estimate). My dilemma is how to quantify this.

From my previous question (and your answer) at:
http://www.experts-exchange.com/Other/Math_Science/Q_26866010.html ,
we know that xv (the variance of average of the two estimates) is minimized when k = av/(av+bv), so maybe this is the answer I'm looking for. It feels right, but I'm just guessing. I'm looking for a more conclusive argument.
------------------------------------------------------

<two different estimates of the same number are both 90% accurate, but they are incredibly different. This isn't really possible>
It's not the estimates that are 90% accurate, it's the confidence levels.
Consider this example: You might be 90% confident that the high for tomorrow in your town will be 70-75F; but if all I know is that you live in the USA, my 90% confidence range would likely be much wider, say 25-90F. Two different estimates of the same number, both 90% confident, but incredibly different.
0
 
LVL 29

Expert Comment

by:leonstryker
ID: 35057475
<Say the first estimate for the amount of fuel for the space shuttle is 35,000 - 36,000 lbs, and the second estimate is 1000 - 100,000 lbs.
Intuitively, if both estimators give answers that are correct 90% of the time, it's a pretty reasonable assumption that the guy with the tighter estimate is more knowledgable than the other guy (he's probably a NASA engineer to be 90% confident of such a tight estimate). My dilemma is how to quantify this. >

Then weight it towards the tighter estimate. K would be based on your level of confidence. In this case if there is a 90% confidence level, then K = .9
0
 
LVL 1

Author Comment

by:Milewskp
ID: 35059249
Hi loen,
<Then weight it towards the tighter estimate. >
Makes sense to me.

<K would be based on your level of confidence. In this case if there is a 90% confidence level, then K = .9>
But both estimates have a 90% confidence level.
My question is how to calculate k that gives the 'best' estimate.  I think that before we can do that, we have to define 'best estimate'.  I think the best estimate might be the one with the tightest range (x1,x2) that has a 90% chance of including the actual value, but I'm open to suggestions.

When you say that k should be .9, what are you using as your definition of 'best estimate'?


0
 
LVL 29

Expert Comment

by:leonstryker
ID: 35071829
<Intuitively, if both estimators give answers that are correct 90% of the time.>
Ok, lets take a step back. If both give correct estimates 90%, then it must follow that the second guys estimate, despite a bigger range, is irrelavent and should not be considered. The only time it would be releavent if it had greater accuracy, so lets say 92% correct vs. the 90% correct. So I would express an equation something like this

k = aA - bA   - assuming the wider estimate is correct more often (exp 92% vs 90%)

xm = (1-k)am + kbm

<A and B have a correlation of 0>
This assumption would have to be disgarded since both are now giving you an estimate on teh same item.


0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 1

Author Comment

by:Milewskp
ID: 35072703
<it must follow that the second guys estimate, despite a bigger range, is irrelavent and should not be considered. >
Note sure if I agree that the wider estimate should not be considered at all. For example, if the two range estimates had the same width and were both 90% confident, I would assume you agree they should be weighed equally, but if one was just 1% wider should it be dismissed? What if it was .1% wider?

<k = aA - bA >
What are a and b?

<A and B have a correlation of 0. This assumption would have to be disgarded since both are now giving you an estimate on teh same item>
Sorry, I don't follow. If you and I give daily estimates for the next day's high temperature, why does that imply the two distributions are correlated?
0
 
LVL 1

Accepted Solution

by:
Milewskp earned 0 total points
ID: 35513303
I think Bayes Theorem is the solution here.
If you have only estimate A, then you assume 90% on the true value being within the range (a1, a2). But when you get additional information, in the form of estimate B, you should use Bayes Theorem to consolidate the two estimates.

For details on how to do this, see the book by Hubbard: How to Measure Anything (last section of Chapter 10).
0
 
LVL 29

Expert Comment

by:leonstryker
ID: 37223820
I've requested that this question be closed as follows:

Closed

for the following reason:

This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.
0
 
LVL 1

Author Comment

by:Milewskp
ID: 37223816
If no objection from the others, please accept my final post as the answer.
0
 
LVL 1

Author Comment

by:Milewskp
ID: 37223821
If no objection from the others, please accept my final post as the answer.
0
 
LVL 37

Expert Comment

by:TommySzalapski
ID: 37224631
You should be able to do that yourself now that the auto-close has been stopped.
0
 
LVL 1

Author Closing Comment

by:Milewskp
ID: 37248624
Thanks everyone for your suggestions.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

In Agile (http://en.wikipedia.org/wiki/Agile_software_development), time and again people ask this question "How would you estimate a release for a product?". When it comes from management they want to know the following: Calculate the man hours wh…
Lithium-ion batteries area cornerstone of today's portable electronic devices, and even though they are relied upon heavily, their chemistry and origin are not of common knowledge. This article is about a device on which every smartphone, laptop, an…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now