Financial Conduct Authority raises questions over TSB's IT regression testing

Giving evidence to the Treasury Committee about the TSB banking IT meltdown, the CEO of the Financial Conduct Authority (FCA), Andrew Bailey, warned that the bank needs proper regression testing.

“Because banks are connected to many external systems, you have to ensure any fix doesn’t cause other problems,” said Bailey. “There are major IT changes taking place on banking IT systems every weekend, such as ring-fencing. What is particular about TSB is it is a complete re-platform of the bank’s system. It is a much bigger scope of project than normal.”

When asked if root causes of the IT problems had been found, Bailey said: “That has inevitably evolved over time. You have a pretty good idea early on, but that understanding evolves. It is not negligent. The rate of emerging faults is falling off.”

He said the FCA’s investigation would be looking at how robust and rigorous TSB’s regression testing across the whole of its IT architecture was. “They [TSB] are in the business of putting in fixes, but something occurs somewhere else and that suggests not enough robust regression testing was done.”

While TSB has a rolling plan to fix IT faults, the FCA admitted it had not seen an overall plan. “There is lots going wrong in branches, like the system connecting to the cash dispenser. These things still need fixing,” said Bailey.

The banking failure occurred as a result of TSB migrating to a new IT platform. The migration was outsourced to Sabis, the IT division of TSB’s parent, Sabadell Group.

However, the meltdown showed that a rollback from a failed migration was not possible at TSB. “You can outsource work, but you can’t outsource responsibility,” said Bailey. “The accountability and responsibility rests with senior management at TSB.”

In a letter to Nicky Morgan, chair of the Treasury Committee, Bailey said the severity of the incident at TSB meant it had required specialist support from IBM to help restore its platforms to pre-migration levels.

During the hearing, TSB’s CEO, Paul Pester, discussed IBM’s findings, stating to the committee that IBM identified problems with applications, middleware and networking as the main causes for the downtime.

Bailey said: “Sabis was the developer, utilising Sabadell’s banking platform, but the responsibility rests with TSB management.”

In his letter to Morgan, Bailey said TSB implemented a “specific decision-making process and matrix of responsibilities to oversee and manage migration”.

Using this decision-making process, which he said the FCA reviewed, TSB took the decision to delay migration from November 2017 to April 2018 because it assessed that there were unacceptable levels of risk to customers and its operating model, for instance because of delays in user acceptance testing (UAT).

During the hearing, Bailey said: “You can’t do the process without the user. We have to be clear what management understands by UAT. Control is fundamental. It cannot be a black box.”

When asked if the UAT was conducted by the outsourcer, Bailey said: “I would want a process of assurance. You can hire firms that specialise in this, or use your own internal audit team. Assurance has to be robust.”

Since banks are connected to many external systems, Bailey stressed the importance of carrying out end-to-end testing.