Why do big IT projects fail? Part one: The professionals

Yann L'Huillier

Yann L'Huillier is group CIO at financial services giant Compagnie Financiere Tradition. Before his current role he was CIO at trading exchange Turquoise where, in eight months, he delivered the complete set of IT solutions, procedures and processes for the exchange to go live in August 2008. Before that, he was CIO at the Boston Stock Exchange (2003-2007), where he designed and launched the Boston Equities Exchange trading platform, and prior to that he spent seven years at the Toronto Stock Exchange where he developed the new Equity Trading Systems.

He says: "Big projects can't be completed on budget or on time. But it's not only IT projects - some of the big architecture/engineering projects fail. You will find some common reasons to all. The first one is that even though most budgets are done well from the start, to be approved they must be scaled down, contingencies removed, to a level where they will receive the blessing of the deciders/payers. Once they are approved there is nothing left for unexpected events (Murphy's law, for example).

"The second reason is that between the time the decision is taken, the project starts and is completed, the situation will change several times. The economics, regulatory [environment], and competition don't freeze at the start, so big projects need a wider scope than the original one. One should plan for unknown changes and have enough buffer in financing and scheduling to account for anything that will go wrong.

"Lastly, when projects are delayed, the wrong solutions are often used, such as increasing staffing instead of de-scoping or phasing the deployment. 'Design is a democracy but implementation is a dictatorship'."

James Martin

James Martin is the former IT chief operating officer (COO) Europe at investment bank Lehman Brothers. Over the past 17 years he has worked for several retail and investment banks, often in the IT COO role. He has been involved in hundreds of large IT projects, including three global Year 2000 programmes.

He says: "In my experience, the key reasons for IT project failure have been consistent across firms and around the world. My top five pitfalls are: Lack of robust business requirements at the outset, leading to unrealistic IT project budgets and timescales; business sponsorship and participation start off strong and then tail off, leaving the IT project drifting; red herring stakeholders frustrating a project by raising numerous side issues and minor concerns; the world outside moving on, which forces a project to be redefined during its course so it never really ends, but just runs out of steam; and the administrative burden imposed on the IT team eats more resource than technical development work.

"The large projects which have been most successful tended to be externally visible to customers, regulators, the public and media. I've also seen 'best practice' lead to 'worst result' projects far too often and I believe that is the root cause of the problem: process has greater emphasis than outcome, and that is not going to get a project over the line."

Anthony Finkelstein

Anthony Finkelstein is professor of software systems engineering at University College London (UCL) and dean of UCL Engineering. Active in industry consulting, he is a fellow of the Institution of Engineering and Technology (IET) and the British Computer Society (BCS).

He says: "The Standish Group in its CHAOS report cites reasons such as: inadequate user involvement; unclear business objectives; failure to control scope; poor architecture; requirements volatility; unsystematic development process; and unreliable estimates.

"None of these would come as a surprise to any IT professional."The more interesting challenge is, therefore, why do we appear to be bound to repeat these same mistakes? I would argue that the problems are more fundamental and not essentially technical - they lie in governance. That is, in the structure of relations and incentives that bind together the business and IT functions of an organisation.

"An organisation with a flawed governance structure cannot articulate its requirements, charter a project, identify appropriately skilled staff, manage the concomitant change process, determine if the project has been successful or even deal with the consequences of failure. Governance is becoming more complex as we increasingly have organisations with federated and outsourced business structures. We will neither understand project failure nor be able to address the causes until we have addressed governance."

Brian Randell

Brian Randell is emeritus professor and senior research investigator at the School of Computing Science at Newcastle University. He was a member of the group of academics who became concerned about the UK NHS NPfIT. From April 2006 until September 2010 he edited the evolving dossier documenting concerns about the programme.

In response to Computer weekly's question, he sent this excerpt from his report, A Computer Scientist's Reactions to NPfIT (Journal of Information Technology, Macmillan, 2007): "The NHS's huge NPfIT project, intended to serve 40,000 GPs and 300-plus hospitals, was claimed to be the world's largest civil IT project. In fact, its ill-fated intended central core, a nationwide electronic health records (EHR) facility, dramatically illustrates one of the most serious causes of large IT project failures.

"The system of systems that was to provide electronic health records was initially designed by a large central team, and intended as a complete "big-bang" replacement for the many and varied existing EHR systems. It would have been far better to employ evolutionary acquisition, ie. to specify, implement, deploy and evaluate a sequence of ever more complete IT systems, in a process that was controlled by the stakeholders who were most directly involved, rather than by some distant central bureaucracy. Authority as well as responsibility should have been left, from the outset, with hospital and general practitioner trusts to acquire IT systems that suited their environments and priorities - subject to adherence to minimal interoperability constraints - and to use centralised services as, if and when they chose."

Philip Virgo

Philip Virgo is secretary general at the Information Society Alliance. He has nearly 40 years' experience of IT projects.

He says: "Large projects nearly always fail unless broken into components that can be delivered step-by-step by mixed teams of users and technicians who know what they have to achieve together and what their next job will be if they succeed - and that they cannot move onto it until they have delivered this one to the satisfaction of the business. That satisfaction may well be an evolving mix of specification, time and budget, because incremental implementation and market change leads to changing expectations and needs.

"My first major project was the merger and decimalisation of the sales ledgers for the companies that had come together to form ICL. My reward was two years at the London Business School, including a course on programme management led by one of the Polaris team. I have watched successes and failures over nearly 40 years. The reasons have not changed. By far the biggest IT project in the UK was the transition of the payment clearing system (PACS), including the ATMs in every high street, to internet protocols and common card standards. It took nearly 10 years to complete all the incremental changes. No one has ever heard of it because success is boring."

Tony Collins

Tony Collins is an investigative journalist who has specialised in large IT projects. He is co-author of Crash, a book on IT disasters, and co-founder of Campaign4Change, which seeks reforms in the public sector. He spent 21 years at Computer Weekly.

He says: "Every project is different, but these are some general lessons:

1. Projects with realistic budgets and timetables tend not to be approved.

2. The more desperate the situation the more optimistic the progress report.

3. Users are likely to reject any system that gives them what they asked for. Better for the project managers to understand what users do rather than what they say they do.

4. CEOs who know a great deal about computer projects are dangerous. The over-confident CEO may try to do too much, too soon and with too little - and possibly remove potential savings from business budgets before the savings are actually made. He gets few warnings because he is surrounded by people who agree with him.

5. Keep it small and simple. If it has to be big, split it into components that are each useful in themselves.

6. A failing project has benefits that are always spoken of in the future tense.

7. In the public sector the unnecessary secrecy over the progress or otherwise of major projects such as Universal Credit continues. It's a pity because it increases the chances of failure."