Friday, October 7, 2011

Failing at Failing: Why IT Will Continue to Suck At Large Projects

Air Florida's Flight 90 departs Washington DC's National Airport in poor weather, and four miles from departure splashes into the Potomac River killing 78 souls.  In response to this event, Air Florida flight operations springs into action by standing up an Investigation Management Organization (IMO) to explain or salvage the failed flight. While no person in the IMO has ever captained a successful flight, each speaks with authority and has demonstrated loyalty to Air Florida management.  Several months pass before the IMO presents its findings and remediation plan to Air Florida's board.  Key amongst their findings, the IMO has determined that the flight wasn't a disaster after all, but a limited success! Consider: the mistakes leading to the crash were forgivable given the circumstances (high complexity, factors beyond corporate control), partial service was in fact accomplished (320 seat miles were delivered), the aging aircraft and staff were disposed of only slightly ahead of schedule and below planned cost (public rescue covered the expense), demand for the route is firmly established (all seats were sold even in blizzard conditions), and new flight management is now prepared to resume service (this time with better union and vendor contracts).

Sound familiar?  Probably not.  But, if we replace the nouns related to flight with nouns related to delivery of large Information Technology (IT) projects then this becomes a boringly typical story.  Standard procedure for handling an IT failure must first and foremost save stakeholder skin.  To do this the team in charge must quickly reframe expectations to allow the disaster to be viewed as a limited victory, replace a few mid-level managers "quietly" to pay lip service to accountability, blame the outcome on something intangible or inscrutable to defuse inquiry, and then issue new contracts and reboot the project, often under a different name to hide the trail of failure.  Because nothing is learned from the failure, the next attempt at the project often yields exactly the same result.  In fact with only 1 in 6 large IT projects actually meeting expectation-time-budget, IT failure is so routine and so poorly handled one could say that IT fails at failing.

Ironically, the large consulting firms that tend to run these projects (er, failures) know this lifecycle so well that their management practices are designed to inoculate against blame, i.e., to hide the facts.  Their playbook calls for billing handsomely during the early years, but delivering nothing of note (except hard-to-validate deliverables like plans, processes, architectures, and environments).  During this time, the team celebrates many meaningless milestones in order to capture key stakeholders on record declaring that the program is making great progress and is on schedule and budget.  When failure eventually becomes evident, all stakeholders are bound together in culpability for misrepresenting progress lo these years. As part of the conspiracy, the leading Business Schools support the marauders' view that these failures can't be prevented (http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar). The only choice for stakeholders is to be part of the coverup or to be blamed.

IT needs to change.  As a profession, we must examine failure accurately and publicly to reduce the chance of future failures.  This change can't come from the private sector directly, which already has reasonable fiscal accountability and relishes secrecy for competitive reasons, but it could come from government.  Governments are accountable to more than their organization (i.e., taxpayers) and suffer an inordinately large number of spectacular failures.  Imagine the equivalent of the National Transportation Safety Board (NTSB) for software - a National Software Success Board (NSSB).  If staffed by active professionals focusing on empirical evidence and adhering to transparency, within ten years the industry outcomes for software projects would look very different.  The 1 in 6 ratio for success would be inverted to 5 in 6, the largest software projects would be one tenth the size of today, a growing body of fact-based knowledge would guide new projects toward success, and buyers would be more able to distinguish the silver-tongued impostors who run many of their projects today from the seasoned professionals who could ensure success.