Fujitsu bosses knew about Post Office Horizon IT flaws, says insider

Largest commercial IT system in Europe

When it was rolled out, Horizon was described by Fujitsu as the “largest non-military IT system in Europe”. About 40,000 Horizon terminals were installed in all Post Offices across the UK. The user interface was a touchscreen and keyboard linked to a PC under the counter which ran on the Windows NT operating system. Branch PCs were connected via ISDN to a back-end mainframe. The Fujitsu-designed Epos software on the PCs was written onto an off-the-shelf system called Riposte.

Our source said the big flaw in Horizon was the way data was being written to Riposte.“Riposte wasn’t really a database, it was a messaging system based on an XML structure where you write messages down into the message store, and then Riposte took care of replicating them,” he said.

“The first thing that you should always do with a system like that is design and agree a data dictionary and a message library repository, basically to say: these are the messages that are allowed to be written to the message store and they all provide the following function.

“It’s almost like an API [application programming interface] so that you have a list of allowed messages that can all be written to the correct format with the correct content.

“You should also have a layer of software that lies on top of the message store that checks that any application above it which is trying to write a message, conforms to the agreed data dictionary. Otherwise, you can just write freestyle to the message store, which is what they were doing. There was no application interface in there, no agreed data catalogue or anything.”

Computer Weekly also spoke to a former Fujitsu employee who worked in the Horizon Service Support Centre from 2001. The support engineer – who also wants to remain anonymous – recognised this new description of Horizon’s badly built message store, adding: “Our job was to fix these problems as they arose. We all knew the code wasn’t fit for purpose and needed rewriting. The data dictionary was still being added to when I got there.”

Senior managers were aware

The most serious allegation raised by the developer is that senior managers at Fujitsu were aware that an important element of the Horizon system did not function correctly and could not be fixed.

For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.

“The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.

“A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”

He gave an example of a message stored previously when a customer bought a stamp. It was feasible that a new message for buying a stamp weeks later could be slightly different.

“When the cash count came along, it found a message it was not expecting and either ignored it, tripped up, or added something it shouldn’t be adding,” he said.

In 2015, Computer Weekly reported another anonymous source who identified the cash writing program as a possible cause of serious problems. He told us the Post Office was warned about the risk of data corruption on the bespoke asynchronous communication system which sent messages between branches and the central Horizon set-up.

Speaking to Computer Weekly in 2015, the anonymous source told us: “The asynchronous system did not communicate in real time, but does so using a series of messages that are stored and forwarded, when the network connection is available. This means that messages to and from the centre may trip over each other. It is perfectly possible that, if not treated properly, messages from the centre may overwrite data held locally.”

Four years later, former Fujitsu engineer Richard Roll wrote in a witness statement to the High Court: “The issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at branch level.”

Roll’s evidence, which was accepted by the judge, suggests that the problems with Horizon identified by our source had not been dealt with by the time the system went live.

Other experts familiar with Horizon that Computer Weekly approached have also supported the developer’s claims.

The developer said he made his superiors at Fujitsu aware of the extent of the Epos system problems, telling them explicitly that the cash account needed to be scrapped.

“I broke it down and said: you can keep these bits at a push if you have to,” he said. “But that bit in the middle, these bits of the engine, the gearbox, you need to throw them away and rebuild them. Starting with the cash account. You’ve got to throw the cash account away and you’ve got to rewrite it.”

Bone of contention

Our source said Fujitsu’s unwillingness to deal with this central problem became a bone of contention with him. He said he was offered more responsibility on the project but refused unless the cash account was rebuilt from scratch. This, he claimed, resulted in him being moved off the Epos side of the project to become Horizon’s LFS [Logistics Feeder Service] team development manager.

He concluded: “It was a prototype that had been bloated and hacked together afterwards for several years, and then pushed screaming and kicking out of the door. It should never have seen the light of day. Never.”

Computer Weekly contacted Fujitsu, asking for a response to the developer’s allegations. A spokesperson for the company said: “As a long-term partner to UK public and private sector organisations, we are dedicated to supporting our customers, our employees and the people they serve in the UK. We provided detailed responses to all questions raised by the House of Commons BEIS Select Committee and are continuing to cooperate with the ongoing Post Office Horizon IT inquiry.”

The Post Office said it would not be appropriate to comment on individual allegations outside of the current independent inquiry and the courts.

The Department for Business, Energy and Industrial Strategy (BEIS), which holds government responsibility for the Post Office, said it had “established an independent, judge-led inquiry to ensure that lessons are learned, and that concrete changes take place at Post Office Ltd. If anyone has relevant information, we would encourage them to provide it to the Post Office Horizon IT inquiry.”

The Williams inquiry into the Post Office Horizon IT scandal is accepting evidence until 25 February and is due to report in the summer.

A Computer Weekly investigation in 2009 first revealed that subpostmasters, who run Post Office branches, were being blamed for unexplained financial losses, which they claimed were caused by errors made by the Horizon system. The Post Office denied this, and many subpostmasters were subsequently prosecuted for theft and false accounting, with prison sentences, community service, criminal records and heavy fines among the injustices they suffered as a result.

It has become one of the biggest miscarriages of justice in UK history.