turtl3 - Fotolia
Botched software update to blame for Post Office Horizon crash
Post Office says planned firmware update caused the problem that left branches unable to do business for 90 minutes
A fault during a scheduled software update caused a UK-wide crash of the Post Office’s controversial Horizon IT system earlier this month.
On 5 November, Post Office branches were unable to do business when the Horizon retail and accounting system crashed. The Post Office blamed Fujitsu for the outage, which it said lasted 90 minutes. Some subpostmasters reported being offline for much longer.
When asked by Computer Weekly for an explanation in the immediate aftermath, Fujitsu hid behind customer confidentiality and a spokesman said: “We are not in a position to comment on anything.”
The Post Office was then criticised for giving subpostmasters a vague explanation for the problem, which experts described as “pointless”. The Post Office told subpostmasters: “It was a memory issue in devices that support and deliver the overall network across the country, which caused the network to fail for around 90 minutes.”
Computer Weekly contacted the Post Office for a more detailed explanation and was told a planned firmware update had caused the problem.
The Post Office told Computer Weekly: “Following a scheduled change to install a firmware update on a network appliance, a fault was detected at 10:03 on Thursday 5 November, which resulted in a network outage. The firmware was backed out and the service was fully restored at 11:29.
“We apologise to all postmasters and customers affected and we are identifying additional steps to ensure this problem doesn’t happen again.”
Post Office has not explained whether the update was performed by Fujitsu or its own staff.
One IT expert commented: “I don’t think the memory comment makes sense other than the firmware update is software that would have been loaded into a device and stored in some kind of memory. Backing out the firmware suggests the memory itself is OK and the new firmware was the problem.”
The Horizon system, used in more than 11,000 Post Office branches, is at the centre of huge scandal.
A Computer Weekly investigation in 2009 revealed that subpostmasters, who run Post Office branches, were being blamed for unexplained financial losses, which they claimed were caused by errors made by the Horizon system. The Post Office denied this, and many subpostmasters were subsequently prosecuted for theft and false accounting, with prison sentences, community service, criminal records and heavy fines among the injustices they suffered as a result. It has become one of the biggest miscarriages of justice in UK history (see timeline below).
Part of the problem was the Post Office’s insistence that Horizon was robust and its denial that Horizon errors were causing unexplained accounting shortfalls. In the multimillion-pound Horizon group litigation, in which subpostmasters were proved right that Horizon was causing shortfalls, High Court judge Peter Fraser described the Post Office’s stance as “amounting to the 21st century equivalent of maintaining that the Earth is flat”.
Referring to the latest Horizon problem, one industry source said: “The good news, for subpostmasters, is that these sorts of network-wide interruptions are so obvious that any resultant shortfalls are likely to be relatively easy to notice and also more likely to elicit a sympathetic response from the Post Office. Contrast that with the scepticism and even hostility with which branch reports of one-off mysterious shortfalls have been met by the Post Office in the past.”
Timeline of the Post Office Horizon case since Computer Weekly first reported on it in 2009
- May 2009: Bankruptcy, prosecution and disrupted livelihoods – postmasters tell their story.
- September 2009: Postmasters form action group after accounts shortfall.
- November 2009: Post Office theft case deferred over IT questions.
- February 2011: Post Office faces legal action over alleged accounting system failures.
- October 2011: 85 subpostmasters seek legal support in claims against Post Office computer system.
- June 2012: Post Office launches external review of system at centre of legal disputes.
- January 2013: Post Office admits Horizon system needs more investigation.
- January 2013: Post Office announces amnesty for Horizon evidence.
- January 2013: Post Office wants to get to bottom of IT system allegations.
- June 2013: Investigation into Post Office accounting system to drill down on strongest cases.
- July 2013: Post Office Horizon system investigation reveals concerns.
- October 2013: End in sight for subpostmaster claims against Post Office’s Horizon accounting system.
- October 2013: Former Lord Justice of Appeal Hooper joins Post Office Horizon investigation.
- November 2013: 150 subpostmasters file claims over “faulty” Horizon accounting system.
- September 2014: Fresh questions raised over Post Office IT system’s role in fraud cases.
- December 2014: MPs blast Post Office over IT system investigation and remove backing.
- December 2014: Why MPs lost faith in the Post Office’s IT investigation, but vowed to fight on.
- December 2014: MPs to debate subpostmaster IT injustice claims.
- December 2014: MP accuses Post Office of acting “duplicitously” in IT investigation.
- January 2015: MPs force inquiry into Post Office subpostmaster mediation scheme.
- January 2015: Post Office faces grilling by MPs over Horizon accounting system.
- February 2015: Post Office CIO will talk to any subpostmaster about IT problems, promises CEO.
- March 2015: Post Office ends working group for IT system investigation day before potentially damaging report.
- March 2015: MPs seek reassurance over Post Office mediation scheme.
- March 2015: Retiring MP aims to uncover truth of alleged Post Office computer system problems.
- April 2015: Post Office failed to investigate account shortfalls before legal action, report claims.
- April 2015: Criminal Courts Review Commission set to review subpostmasters’ claims of wrongful prosecution.
- May 2015: IT system related to subpostmaster prosecutions under review by CCRC.
- June 2015: Post Office looking to replace controversial Horizon system with IBM, says MP.
- July 2015: Campaigners call for independent inquiry into Post Office Horizon IT system dispute.
- October 2015: James Arbuthnot takes Post Office IT fight to House of Lords.
- November 2015: The union that represents Post Office subpostmasters has warned of a problem with the Horizon accounting system.
- November 2015: An email from Post Office IT support reveals a problem with the Horizon system and supporting processes that could lead to accounting errors.
- November 2015: Group litigation against Post Office being prepared in Horizon dispute.
- February 2016: Post Office faces group litigation over Horizon IT as subpostmasters fund class action.
- June 2016: Post Office chairman Tim Parker says there would be “considerable risk” associated with changing its Horizon computer system.
- November 2016: The legal team hired by a group of subpostmasters will take their case to the next stage.
- January 2017: The group action against the Post Office that alleges subpostmasters have been wrongly punished for accounting errors gets a green light from the High Court of Justice.
- March 2017: 1,000 subpostmasters apply to join IT-related group litigation against Post Office.
- April 2017: Investigation into claims of miscarriages of justice in relation to a Post Office accounting system has appointed a forensic accountant firm.
- May 2017: Hundreds of subpostmasters have applied to join IT-related legal action since March.
- July 2017: Post Office defence in computer system legal case due this week.
- August 2017: Campaigners submit initial evidence in group litigation against Post Office over controversial Horizon IT system.
- October 2017: Subpostmasters’ group action against the Post Office reaches an important milestone.
- November 2017: An end is in sight for subpostmasters’ campaign against alleged wrongful prosecution, which they blame on a faulty computer system.
- November 2017: The High Court judge managing the subpostmasters versus Post Office legal case over an allegedly faulty computer system tells legal teams to cooperate.
- January 2018: Forensic investigation into Post Office IT system at centre of legal case nears completion.
- April 2018: Criminal Cases Review Commission forensic examination of the IT system at the centre of a legal case against the Post Office has raised further questions.
- May 2018: Post Office branches unable to connect to Horizon computer system for several hours after morning opening time.
- October 2018: After over a decade of controversy, next week marks the beginning of a court battle between subpostmasters and the Post Office.
- November 2018: Case against Post Office in relation to allegedly faulty computer system begins in High Court.
- November 2018: High Court case in which subpostmasters are suing the Post Office has revealed a known problem with a computer system at the core of the dispute.
- November 2018: A High Court trial, where subpostmasters are suing the Post Office for damages caused by an allegedly faulty IT system, ends second week.
- November 2018: Post Office director admits to Horizon errors and not sharing details with subpostmaster network.
- November 2018: The High Court trial in which subpostmasters are suing the Post Office has reached an important stage.
- December 2018: CCRC may hold off subpostmaster decision until after Post Office Horizon trial.
- December 2018: Court case where subpostmasters are suing the Post Office set to span at least four trials and extend into 2020.
- January 2019: Subpostmasters’ campaign group attacks Post Office CEO Paula Vennells’ New Year honour amid ongoing court case.
- January 2019: Thousands of known errors on controversial Post Office computer system to be revealed.
- March 2019: Tech under spotlight at High Court in second subpostmasters versus Post Office trial.
- March 2019: Post Office considered Horizon IT system “high-risk”, court told.
- March 2019: CCRC watching Post Office Horizon trial closely.
- March 2019: Judge rules that Post Office showed “oppressive behaviour” in response to claimants accused of accounting errors they blamed on Horizon IT system.
- March 2019: Post Office “lacked humanity” in the treatment of subpostmasters, says peer.
- March 2019: A High Court judge heard that the Post Office did not investigate a computer system error that could cause losses, despite being offered evidence.
- March 2019: The Post Office legal team in the case brought by more than 500 subpostmasters has called for the judge to be recused after questioning his impartiality.
- March 2019: A senior civil servant asked the Post Office to repay public money it had wrongly allocated to paying legal costs.
- April 2019: Subpostmaster claimants’ legal team makes application for the Post Office to pay millions of pounds of costs associated with trial.
- April 2019: Post Office to appeal judgment from first Horizon trial.
- April 2019: The Post Office’s claim that the judge overseeing the case concerning its controversial Horizon IT system was biased has been dismissed – but will now be considered by the Court of Appeal.
- April 2019: MP questions government over Post Office Horizon case.
- April 2019: Government says no conflict of interest in trial despite Post Office chairman’s dual role.
- May 2019: The Court of Appeal has refused the Post Office’s application to appeal a major decision in the Horizon IT trial.
- May 2019: The Post Office has applied for permission to appeal judgments from the first trial in its IT-related legal battle with subpostmasters.
- May 2019: The judge in the Post Office Horizon trial has ordered the organisation to pay the legal costs of its courtroom adversaries, and refused to give permission to appeal a major judgment.
- June 2019: Post Office asks Court of Appeal for permission to appeal judgment in first Horizon trial.
- July 2019: The Post Office has admitted that some subpostmasters are at risk of accounts not balancing due to an error it does not understand.
- July 2019: Problem revealed during High Court trial left subpostmaster with £18,000 surplus after IT system failed to register full amount of cash scanned in.
- August 2019: Subpostmasters suffering slow running and frozen terminals while Post Office searches for a fix to issues apparently caused by a software update.
- August 2019: The Post Office has fixed the latest problems with its Horizon system, affecting hundreds of branches.
- October 2019: A High Court judgment for a trial that focused on the Post Office’s IT system at the centre of a multimillion-pound litigation will be announced early next month.
- November 2019: The Court of Appeal has rejected a Post Office application to appeal judgments made in its multimillion-pound battle with subpostmasters over IT system failures.
- November 2019: Peer calls for clear-out of Post Office board after Court of Appeal confirms major court defeat.
- December 2019: The Post Office has settled its long-running legal dispute with subpostmasters, and will pay £57.75m in damages.
- December 2019: Subpostmasters ended their legal battle with the Post Office at the optimal time, according to the lawyer that managed the High Court action.
- December 2019: Subpostmasters proved right on IT system failures as calls for full public inquiry mount.
- December 2019: Criminal Courts Review Commission to review Horizon judgment “swiftly”.
- December 2019: National Federation of Subpostmasters cries foul after court ruling on controversial computer system.
- December 2019: Former Post Office CEO apologises to subpostmasters over Horizon scandal.
- December 2019: Call for former Post Office CEO to step down from public roles after IT court battle lost.
- January 2020: Fujitsu must face scrutiny following Post Office Horizon trial judgment.
- January 2020: Subpostmaster group calls for government to pay legal costs for Horizon trial.
- January 2020: Why subpostmasters are calling on the government to pay Horizon trial costs.
- January 2020: Department for Business, Energy & Industrial Strategy says it did not make decisions in the Post Office’s recent court battle.
- January 2020: Government should not be allowed to dismiss subpostmasters’ claims over Horizon IT scandal.
- January 2020: Police sent information about potential Fujitsu staff perjury in subpostmaster prosecutions.
- January 2020: Prosecutions are a significant step closer to being sent to the Court of Appeal as Criminal Courts Review Commission forms a group of commissioners to review them.
- January 2020: Alan Bates: The “details man” the Post Office paid the price for ignoring.
- February 2020: The government has refused to pay the huge legal costs subpostmasters incurred in their battle with the government-owned Post Office, which they won.
- February 2020: Members of Parliament seeking a public inquiry into the Post Office Horizon scandal face huge challenges, but pressure and time could force justice.
- February 2020: Calls for inquiry into Post Office IT scandal increase in Parliament, with cross-party support.
- February 2020: Care Quality Commission to review concerns over Paula Vennells’ appointment after they were raised by a former NHS consultant psychiatrist.
- February 2020: Government admits it was too passive managing Post Office as parliamentary pressure builds.
- February 2020: Minister says Post Office IT experts misled the government when it asked questions about subpostmasters’ concerns over Horizon IT system.
- March 2020: Boris Johnson commits to “getting to the bottom of” Post Office Horizon IT scandal.
- March 2020: Boris Johnson’s commitment to inquiry into Post Office scandal in doubt.
- March 2020: MPs call on PM to commit to full public inquiry into Post Office Horizon IT scandal.
- March 2020: Those who did not play by the rules in Post Office Horizon scandal “should face prosecution”.
- March 2020: MPs told to hold to account those responsible for Post Office Horizon IT scandal.
- March 2020: The Post Office has sparked anger with secret settlements with subpostmasters outside the recent legal action against it.
- March 2020: Labour MP Karl Turner tells Computer Weekly that the Post Office Horizon scandal is the most grotesque version of predatory capitalism he has ever seen.
- March 2020: MP Kevan Jones has warned a government minister not to repeat the mistakes of predecessors in relation to the Post Office Horizon IT scandal.
- March 2020: Criminal Cases Review Commission to use Microsoft Teams to ensure review of subpostmaster prosecutions is held on time.
- March 2020: Post Office postpones subpostmaster compensation scheme amid Covid-19 crisis.
- March 2020: Meeting reviewing subpostmaster applications to appeal criminal prosecutions moves into second day.
- March 2020: Subpostmaster prosecutions to be considered by Court of Appeal for miscarriages of justice.
- March 2020: How subpostmasters made legal history with biggest referral of potential miscarriages of justice.
- April 2020: Met Police examines information about evidence given in court by Fujitsu staff on the Horizon IT system.
- May 2020: Subpostmasters who had their lives ruined by the Post Office’s faulty IT system have received their damages after a High Court victory.
- May 2020: A senior Post Office executive at the centre of an IT scandal, who tried to mislead a High Court judge in relation to it, has left the organisation without fanfare despite many years of service.
- May 2020: Post Office re-examines hundreds of prosecutions that could have resulted from faults in Horizon IT system.
- June 2020: A campaign group representing subpostmasters wrongly prosecuted for theft and false accounting by the Post Office is raising money to help clear the names of victims of the scandal.
- June 2020: Subpostmasters to force scrutiny of government’s role in Post Office IT scandal.
- June 2020: The Criminal Cases Review Commission sends 47 more subpostmaster cases to Court of Appeal and asks government to review private prosecution powers.
- June 2020: Select committee chair writes to former Post Office CEO demanding answers over her role in IT scandal.
- June 2020: The government has been accused of launching a review that fails in getting to the bottom of one of the biggest miscarriages of justice in UK history.
- June 2020: Subpostmasters will not cooperate with government review into IT scandal.
- June 2020: The government’s proposed review of the Post Office IT scandal has received a further setback as forensic accountants join subpostmasters in refusing to back it.
- June 2020: Call for government review of Post Office Horizon scandal to have the power to force individuals to give evidence under oath.
- June 2020: Subpostmasters seeking justice in the Post Office Horizon IT scandal are regaining momentum in Parliament.
- June 2020: Healthcare regulator will be discussing concerns about former NHS boss chairing an NHS trust at an upcoming meeting.
- June 2020: Second Sight is working with law firm in appeals by subpostmasters against criminal convictions in Horizon IT scandal.
- June 2020: Post Office and Fujitsu blame each other for many of the failings in the Horizon IT scandal that wrecked lives.
- June 2020: Parliamentary Justice Committee to hold short inquiry into the rules and regulations surrounding private organisations’ ability to initiate criminal proceedings.
- July 2020: Victims of the Post Office Horizon IT scandal need to raise thousands of pounds in a week or those responsible for their suffering will avoid scrutiny.
- July 2020: The government is set to face scrutiny over its involvement in the Post Office Horizon IT scandal, described as one of the biggest miscarriages of justice in modern UK history.
- September 2020: The government repeats that it won’t pay victims’ legal costs and confirms review into the scandal will not have the power to call witnesses.
- September 2020: Subpostmasters still not being told about all the known errors in the controversial Post Office branch accounting and retail system that they use.
- October 2020: The Post Office has chosen not to contest 44 out of 47 appeals, meaning most are likely to have their names cleared, but others still face a Court of Appeal battle for justice.
- October 2020: MPs are demanding the government holds a full statutory public inquiry into the Post Office IT scandal.
- October 2020: NHS regulator continues enquiries about the appointment of former Post Office CEO at Imperial College Healthcare NHS Trust as more damning details emerge.
- October 2020: Government minister met with former subpostmaster online in an attempt to get victims of the Post Office Horizon scandal involved in government review.
- October 2020: The Post Office is focusing urgently on fixing an IT error suffered by a subpostmaster amid the ongoing IT scandal.
- October 2020: Labour politicians are calling for the government to give the Post Office Horizon scandal inquiry the power to force witnesses to give evidence if they don’t cooperate.
- October 2020: Imperial College Healthcare NHS Foundation Trust has asked for external review of its process when appointing controversial executive.
- November 2020: Government faces scrutiny of its handling of the Post Office IT scandal that destroyed subpostmasters’ lives and livelihoods.
- November 2020: Post Office branches offline during busy business hours after suffering an IT error that the Post Office said related to IT from supplier Fujitsu.
- November 2020: Fujitsu is refusing to explain what caused a national system outage in Post Office branches last week, despite the Post Office confirming the issue was the fault of the supplier.
- November 2020: The Metropolitan Police opens criminal investigation into Fujitsu staff who gave evidence in trials of subpostmasters wrongly prosecuted and even imprisoned for financial crimes.
- November 2020: Post Office criticised over vagueness of its explanation of the cause of a UK-wide IT failure that saw subpostmasters unable to do business.
Read more on IT for retail and logistics
-
![]()
Post Office boss said subpostmasters had hands in till and blamed technology for missing cash
By: Karl Flinders
-
![]()
Further extension to controversial Post Office contract with Fujitsu inevitable
By: Karl Flinders
-
![]()
Some former Post Office staff should be jailed over scandal, says government minister
By: Karl Flinders
-
![]()
Fujitsu should pay half of Post Office scandal costs, says select committee chair
By: Karl Flinders
