Software failure paralyses O2’s 4G network

Mobile network operator O2 has blamed a problem with a third-party software installation for a nationwide network outage that has left millions of subscribers unable to access 4G data services on their smartphones.

With 32 million active connections, O2 bills itself as the UK’s most widely used mobile network, which in reality means a number of mobile virtual network operators (MVNOs) run their services over it. As a result, users of GiffGaff, Lycamobile, Sky Mobile and Tesco Mobile have also been affected.

An O2 spokesperson said: “We’re aware that our customers are unable to use data this morning. One of our third-party suppliers has identified a global software issue in their system which has impacted us. 

“Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wi-Fi wherever they can and we apologise for the inconvenience caused.”

The operator said it understands the problem – which began at around 4.45am on 6 December – is affecting operators in a number of other countries besides the UK. The Japan Times has reported that services on the SoftBank network are also down in parts of Japan, with the outage beginning at 1.39pm local time, the same time as O2's.

No other UK operators, however, have been affected, and voice calls over the O2 network can be made as normal. There is also no indication yet that the outage is due to any kind of malicious activity or cyber attack.

O2 is encouraging customers to keep track of its service status website in the meantime.

The issue has also hit a number of other organisations that rely on O2’s network, including Transport for London (TfL), which reported that its bus information display boards had stopped working at approximately 5am.

“The company that updates the data to our network of Countdown Systems is currently experiencing difficulties in providing this service. This issue is being investigated,” said a TfL spokesperson on Twitter.

Similarly, Shropshire Council has reported the same problem with its car park payment machines, which also rely on O2 data connections.

While a number of O2 users have expressed their frustration on social media, calling for compensation from the operator, official guidance from regulator Ofcom suggests they will be disappointed.

Compensation arrangements are highly dependent on circumstances and customers may be entitled to additional refunds or account credits in extreme cases where repairs take longer than anticipated. At the time of writing it is unlikely that the current period of downtime would qualify, meaning any redress for O2 users would be entirely at the operator’s discretion.

Update: 4,00pm, Thursday 6 December 2018

As of 4.00pm on Thursday 6 December, almost 12 hours since the outage began, the third party software issue had been identified as originating at Ericsson. O2 said its teams continued to work closely with its supplier to fix the problem.

In a brief statement on Twitter, Mark Evans, CEO of O2 parent Telefonica UK, said: "I want to reassure our customers that we are doing everything we can to fix the issue with our network and say how sorry I am to everyone affected. My teams are working really hard with Ericsson to find a swift resolution."

Anecdotally, some O2 users have now reported that they are unable to make or receive calls, or send SMS messages.

Update: 5.00pm

Networking supplier Ericsson has now issued a statement saying it has identified issues in certain nodes on the core network, which led to outages for customers using two specific versions of the Serving GPRS Support Node - Mobility Management Entity (SGSN-MME).

The supplier's analysis indicates that the specific problem was an expired certificate in the software installed with customers such as O2 and SoftBank.

“The faulty software that has caused these issues is being decommissioned and we apologise not only to our customers but also to their customers. We work hard to ensure that our customers can limit the impact and restore their services as soon as possible,” said Ericsson president and CEO, Börje Ekholm

Ericsson said that most of the affected network services have now been successfully restored, although at the time of this update, many O2 customers were still saying they were unable to access the 4G network.