Thursday, April 26, 2012

The Cost of Predictability


Predictability is a highly valued property of organizations, in particular, of IT organizations that concurrently run more than a handful of projects. Indeed, managing expectations with stakeholders, coordinating dependencies, budgeting/planning and minimizing the stress and chaos from surprises are good career-enhancing incentives for valuing predictability.

But what does predictability mean in practice? A good working definition is: the perception of a pattern of regularly met commitments. There may be a psychological basis as to what constitutes a pattern in the oft-demonstrated ratio of 4 to 5 "rights" offset every "wrong," but many risk-averse IT organizations use a target figure of 95% of commitments met. 

Of course, the implication is that for meeting commitments 95% of the time, each commitment should have a 95% confidence (probability) of being met. Thus, averaging the results of all commitments over time will hit the portfolio target of 95% of commitments met. 

You must have figured out by now that, except by the very foolish, commitments are based on estimates. Estimates are forecasts of an uncertain future, and the more novel the activity that needs an estimate, the less reliable the estimate, i.e., repetition improves accuracy. But, most IT activities - especially software creation - are relatively unique for the organization making the estimate, so the estimates must be padded (or buffered) to ensure that they have a 95% confidence interval.  Interestingly, most organizations use ROMs (or, Rough Order of Magnitude guess) as an estimating technique, and this approach produces the most padding.

Before we get to our conclusion, three other facts about estimates and commitments are relevant. First, all projects are like a perfect gas, that is, they expand to consume the available resources and never deliver early. Second, information theory tells us that the maximum information yield is when any given estimate has a 50% chance of being exceeded, and a 50% confidence estimate is always smaller (or equal) than a 95% confidence estimate.  And finally, any project that exceeds its estimate, whether estimated at the 50% or 95% confidence level, will experience pressure to conclude quickly.

Now, imagine two portfolios: portfolio Safe demands that all projects estimate to 95% confidence, and portfolio Efficient demands that all projects estimate to 50% confidence. The sum of resources consumed by projects in the Safe portfolio will be higher than the sum of resources consumed by projects in the Efficient portfolio. As the names gave away, a 50% target estimate is more efficient than 95% target.

Phew, we're finally here: the cost of predictability is real, often high, and based on personal observation, in some cases extraordinarily high. Seeking predictability with high confidence estimates may be okay if used as a device to manage dependencies in a large program, when the cost of a single project's blown commitment has a cascading effect on other projects. Or, it may be justified when used to manage expectations in an organization with nasty politics.  But folks concerned with efficiency should adopt a different set of metrics than "on time, on budget," jettison the use of ROMs as an estimating technique, and work on achieving better estimates.