Visa reveals 'rare' datacentre switch fault as root cause of June 2018 outage

Visa has revealed a “rare defect” in a datacentre switch is what stopped millions of credit card transactions from being carried out during its UK-wide outage on Friday 1 June, in a letter to the Treasury Select Committee.

The Committee is understood to have contacted the credit card payments firm, seeking both clarification over the cause of the outage and assurances about what action Visa is taking to prevent a repeat of it occurring at a later date.

Over the course of the 11-page missive, Visa expands on its previous explanation of a “hardware failure” being the cause of the 10-hour outage by laying the blame on a defective switch in its primary UK datacentre, which – in turn – delayed its secondary datacentre from taking over the load.

The primary and secondary datacentre are setup so that either one has sufficient redundant capacity to process all the Visa transactions that take place across Europe should a fault occur, and the systems are tightly synchronised to ensure this can happen at a moment’s notice.

“Each datacentre includes two core switches – a primary switch and a secondary switch. If the primary switch fails, in normal operation the backup switch would take over,” the letter reads.

“In this instance, a component within a switch in our primary data centre suffered a very rare partial failure which prevented the backup switch from activating.”

This, in turn, meant it took longer than intended to isolate the primary datacentre and activate the backup systems that should allow its secondary site to assume responsibility for handling all of the credit card transactions taking place at that time.

The firm’s UK datacentre operations team were alerted to the faulted switch at 2.35pm on Friday 1 June, after noting a “partial degradation” in the performance of the company’s processing system, before initiating its “critical incident” response protocols, the letter continues.

“It took until approximately 19:10 to fully deactivate the system causing the transaction failures at the primary datacentre,” the letter continues.

“By that time, the secondary data centre had begun processing almost all transactions normally. The impact was largely resolved by 20:15, and we were processing at normal service levels in both datacentres by Saturday morning at 00:45, and have been since that time.”

Visa is also quick to point out that at no point during the incident did a “full system outage” occur, but admits the percentage of transactions that were processed successfully did fluctuate, with peak periods of disruption occurring between 3.05-3.15pm and again between 5.40pm-6.30pm.

During these times, around 35% of attempted card transactions failed, but this failure rate dropped outside of these periods to 7%.

“Over the course of the entire incident, 91% of transactions of UK cardholders processed normally; approximately 9% of those transactions failed to process on the cardholders’ first attempt,” the letter continues.