If two software engineers each give me a range estimate for the cost of a software project, say
1) $250K - $260K
2) $100K - $500K
both with a 90% confidence level, how should I reconcile this data to give me a single 90% range estimate?
With a range like $100K-$500K the engineer has obviously decided that it's going to be about $250K and covered his arse by giving a wide margin either side, which he's 90% certain that he won't go over, even if the project gets seriously delayed. He's taking the piss a bit but he won't get accused of underestimating the costs.
For a $250K-$260K range the other engineer either has a very good understanding of the costs and can guarantee no project slippage (very unlikely) - or he's an idiot. Even delivering a few weeks late can drive a project way over $10K extra.
From experience of software projects I'd say that the $100K-$500K estimate would be less likely to be wrong, however the $100K figure is likely to be low, given the other engineer's estimate.
If you can think of any reasonable sized software project that has been delivered on time, within budget and to specification then you've found something extremely unusual. Certainly no government IT project that I can think of has managed all 3 criteria. Actually I'd be surprised if you can find a government project which hit 2 of the 3!
Given all that, the only figures I'd be confident in are the wider range.
AS shown above you cannot reconcile them
However if you assume (a BIG assumption ) that both estimates are the result of normally distributed independent measurements you can treat the numbers as representing two normally distributed random variables and add the distributions.
Folks,
I work in the Project Controls group of my company, and there are/ will be situations where we get multiple estimates for all kinds of projects (IT and other). I need a way to calculate an aggregate range estimate given the range estimates we are provided.
Now, when someone givens me a range estimate with a 90% confidence level, I make the assumption* that 90% of that individual's 90% confidence level estimates will be correct (ie, the actual values will fall within the ranges 90% of the time). My job is not to second-guess their estimates, but to provide management with the 'best' range estimate given the inputs I've received.
*You may argue that this is an unrealistic assumption, but I would direct you to Doug Hubbard's book "How to Measure Anything". Anyway, whether this assumption is realistic or not is not the question. The question is how to calculate an aggregate range estimate.
Any help you could provide would be much appreciated.
0
With monday.comâ€™s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.
If you are going on a 90% rule I think you can also assume that there will be 10% of individuals whose 90% confidence levels are hopelessly wrong 90% of the time - either they don't have the experience or skills to get the numbers right. I have seen this in practice. Out of the approx 20 people that I work with a couple of them will consistently seriously under-estimate how long a programming task will take to do, even though they have been programming for 30+ years!
When the estimates are as far apart as the original examples I think you can reasonably assume that one of them is not right and the job is then to work out which is wrong and why.
If you assume they are normal distributions, then the 250-260 is N(255, 3.0398) and the second is N(300, 121.5914) So the variances are 9.2404 and 14784.46
So if you are adding these, it makes X~N(555, 121.629) which has a 90% confidence interval of
355k - 755k
If you are averaging them, then you would have X~N(277.5, 86.004) or a 90% conf of 136k - 419k
Hi Tommy,
<If you are averaging them, then you would have X~N(277.5, 86.004) or a 90% conf of 136k>
So you are saying that, if I assume noraml distributions, adn I assume that the estimators are correct 90% of the time (ie, the confidence interval really is 90%), then I can assume that the cost in this example will be in the range $355k-755k 90% of the time?
I repeat
However if you assume (a BIG assumption ) that both estimates are the result of normally distributed independent measurements you can treat the numbers as representing two normally distributed random variables and add the distributions.
Hi Tommy,
<Assuming that those things are true, that is how the math works out. >
But, given our assumptions, we already know that the actual cost will be between $250K - $260K 90% of the time, so how can they also be between $136k - $419k 90% of the time? It can't be both.
1. Project efforts are more likely to be approximately log-normally distributed than normally distributed. Determining the variance of a sum of log-normally distributed variables is much messier than determining the variance of a sum of normally distributed variables, but given the slop in these numbers I wouldn't worry about it too much. Use an approximation. "Why measure it with calipers if you're going to cut it with an axe?"
2. Were these price quotes? They were stated in units of currency, after all. If so, they're really not estimates, but proposed commitments, which is a very different thing. Saying $250-$260K amounts to a fixed bid.The two engineers may have estimated the effort as the same, but the second one is much more willing to assume the risk of running over in exchange for pocketing the extra if s/he comes in under.
Much pain and suffering has occurred in our "profession" due to lack of understanding of the difference between a Target, an Estimate, and a Commitment, and the failure to compare any of them to past reality.
0
Featured Post
Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.
For a $250K-$260K range the other engineer either has a very good understanding of the costs and can guarantee no project slippage (very unlikely) - or he's an idiot. Even delivering a few weeks late can drive a project way over $10K extra.
From experience of software projects I'd say that the $100K-$500K estimate would be less likely to be wrong, however the $100K figure is likely to be low, given the other engineer's estimate.
If you can think of any reasonable sized software project that has been delivered on time, within budget and to specification then you've found something extremely unusual. Certainly no government IT project that I can think of has managed all 3 criteria. Actually I'd be surprised if you can find a government project which hit 2 of the 3!
Given all that, the only figures I'd be confident in are the wider range.