CIO interview: Gerry Pennell, CIO, London 2012 Olympic Games

A typical day in the Olympics Technology Operations Centre 

Every day during the Olympics and Paralympics, about 190 people at the Technology Operations Centre (TOC) monitored the IT systems, working on a three-shift pattern, 12 hours at a time. Pennell's day started early.

"My daily pattern was to come in at 6.30am for a morning call with the IOC [International Olympic Committee] about issues yesterday and what we're expecting today, and then after that brief our management on status," he says.

"Then depending on the day, I would typically spend some time in the TOC reviewing what was going on, such as cyber security or the volumes we were encountering on the website, etc. When I could, I would go out to the venues, not to interfere with what the troops were doing, but mainly to encourage them and check people were happy and focused and doing well."

Months of preparation and testing ensured there were no shocks, either internally or for the viewing public.

"We'd planned enough, there was tuning and tweaking going on, but we didn't have a problem. There were no moments of panic," says Pennell.

"When you're running a large technology operation you expect components to fail, and we had some component failures. The classic ones were equipment close to the field of play, in particular commentator terminals when it rained. During the Olympics we had quite a few sudden showers. In those, from time to time a PC would stop working, but we were ready for it and able to do a fast box-swap. We had those kinds of things going on," he says.

The mobile Olympics 

If there was one surprise, it was the level of engagement from mobile phone users. London 2012 was the first Olympics of the smartphone era, and Pennell says he expected the consumerisation of IT to be a new challenge with little previous Games experience to fall back on. But the volumes of traffic generated reflected the scale of the shift in consumer behaviour.

"lt's the speed at which these things are changing, the BYOD [bring your own device] thing, the way the Games has to cope with that trend. It wasn't a surprise, but it came out very strongly in London," says Pennell.

"The most compelling statistic is that at peak moments over half the load on our web servers, which were hosted by BT, was coming from people either accessing the mobile site or using the mobile app. We expected this to be big in London, but I wouldn’t have guessed it would be that big. At peak, on the website there were over half a million concurrent users, so it was pretty hot," he says.

"What was interesting was the interaction between those two things – the combination of live results available and people being able to get it on a mobile basis meant there was more volume of people looking for the latest updates than we had anticipated. We had to tune some things to make it work, which went fine. One of the highest peaks in the website use and mobile use in particular was the Andy Murray tennis final."

The pre-Games doomsayers warned that mobile networks wouldn't cope, but London 2012 organiser Locog had set up the Joint Olympic Operators Group (Joog) to work with the major mobile networks to ensure sufficient capacity.

"The infrastructure challenge was with the 3G networks in London, which were already stretched even without the Games," says Pennell. "[We had to consider how we were] going to provide a reasonable level of service to spectators for access to apps as well as for sharing user-generated content." 

Through Joog, the mobile operators invested in a lot of shared infrastructure.

"We were offering a much better level of mobile networking experience at our venues than would be typical at a normal sporting event, so you could access everything on your device and share pictures and all the rest of it," he says.

"In a similar manner, we worked with BT across the Olympic Park. BT put in the world's largest high-density Wi-Fi system to enable services to be accessed from people's devices, and it all worked very well – my main measure being the absence of complaints. This was the Games when the consumerisation of technology really hit what the Olympics had to deliver."

Daily cyber attacks on Olympics systems 

The other, less positive, expectation was that the Olympics would be an attractive target for hackers. This certainly proved to be the case. Pennell admitted to facing at least one particularly major cyber attack, and while he would not elaborate on the specifics, he says the security challenge was ongoing.

"We were attacked every day. Some of the attacks were fairly well orchestrated. Some just before the Games were automated," he says.

"We prepared for this well in advance so it didn't cause us any problems. It was interesting watching some of it, because some of the groups that were involved, from the hacktivist community, got fed up of trying to take us down and moved their attentions to other organisations. 

"Entertainingly, some of this stuff was remarkably easy to track as there was the hashtag #letthegamesbegin, and they were publishing when they were going to attack – '7pm on Friday night let's have a go at the website'. 

"There were other more serious attacks, but none of it caused a problem, but that's down to the right amount of planning and energy in preparation."

There is still work to do before Pennell leaves. His main focus is preparing for the formal de-brief in November to the organisers of the Rio 2016 Olympics in Brazil. 

Meanwhile, all the equipment is being shipped back to a warehouse in Stevenage before being returned to suppliers or otherwise disposed of. Much of the hardware is being sold off at bargain prices to schools, part of a sustainability strategy that attracted praise from the IOC.

"The IOC commented we were the first technology delivery for a Games to have an integral sustainability strategy. That covered a lot of dimensions, from the energy efficiency of desktops to server virtualisation," he says.

"One of our responsibilities was to organise volunteers to run around the venues with bits of paper delivering results to VIPs and technical delegates. From Athens and Beijing, about 50 million sheets of paper were printed – that's a lot of paper. We cut that down to about 20 million. 

"Plus we have a lot of data now in terms of carbon footprint to set a baseline for future games to refer back to. We're pretty proud of that."

The Olympics in the cloud 

There is another important lesson Pennell hopes the IOC will learn – the potential for using the cloud to deliver the IT for future Olympics.

"The IOC understands my view that the Olympics is absolutely a textbook opportunity for a cloud implementation. It's peaky, with very high volume for very short period of time. From an economic point of view, you want a variable cost base, not a fixed cost base. The model for the Games, whereby you have to commission and build datacentres with vast rows of servers just for a short period of time, doesn't make economic sense," he says.

"The cloud and technology associated with cloud were not mature enough in 2008 for us to have been able to successfully manage that transition, but over the next few editions of the Games – and this is going to have to be driven by the IOC – a transition will have to take place."

But London 2012 was the first to dabble in hosted services and cloud on a small scale.

"A number of our key services were not true cloud, but were hosted externally. Our IT incident management system was a hosted service, our arrivals and departures system was a hosted service, so we did more than previous Games to get to a place where we weren't doing everything in our datacentres," he says.

"In one incidence, the torch relay nomination process, that was based on a true public cloud offering and worked very well, and gave me six weeks of quite significant volumes at minimum cost. We used Microsoft Azure – it worked extremely well, and it was cheap."

Lessons learned from delivering IT for the London 2012 Olympics 

So as Pennell starts to reflect, what are the lessons he has learned, and what could his CIO peers also take away from his Olympics experience?

"A lot of the lessons you learn from a project like this are no different from a big IT delivery in a commercial environment. The big one for me is always the same – if you can get the right team you can work miracles," he says.

"We were very fortunate in Locog, not just in technology, to get some pretty fantastic talent. Perhaps that’s the thing the Olympics more than anything enables you to do. People really want to work on this project, which means you get the opportunity to hire some very good people. It’s a privilege to lead that talented group."

But in briefing his successor, the technology director for Rio 2016, there will be valuable pointers that any CIO can benefit from.

"It's about getting the foundations correct in terms of contracts with providers, and in terms of clarity around scope, because one of the challenges in the Olympics is there's always more you can do. The trick is to work out what you're actually going to do and focus on those things. Make sure your budget is secured and you know what you can do. The fourth key foundation is building your team. If you can get those four things correct then regardless of whether it's the Olympics or any large technology delivery programme, you're in good shape," says Pennell.

At a personal level too, there is plenty for him to look back on.

You go into these things with varying degrees of terror

London 2012 CIO Gerry Pennell

"However much effort you put into testing, and into risk mitigation around cyber security, you never actually know it's going to work properly until you get there," he says.

"You go into these things with varying degrees of terror, especially when people say beforehand that the reputation of the UK technology industry is resting on you Gerry, and other helpful comments like that! You go into it with a nervous mindset. Then when you get to the point it's actually working, the results are being pumped out, scores of organisations are taking that results feed, supporting all the broadcasters, pumping out TV graphics etc. When you see that all working, it's pretty fantastic."

But there is also a definite sense from Pennell and all his team, as they move to new jobs too, of wanting to use their experience to prove to others – in the UK generally, as well as in UK IT – of what is possible if you put your mind to it.

"We started by talking about the general sense of nervousness from various quarters as to whether the UK could successfully deliver a huge technology programme on a massive integration, deployment and operational challenge such as this, with all the different providers involved. If there was one take-away I would want people to hear it would be this: Yes, of course, the UK absolutely has the capability to manage these types of projects well," says Pennell.

"You come away with a sense of just how much can be achieved if you can only get everybody pointing in the right direction and working together. The amount of stuff that's been done in London here is pretty phenomenal. You get a sense of what's possible, an aspirational sense of what you can achieve if you get everybody focused on it."

What next for ex-London 2012 CIO Gerry Pennell? 

So if this was the job of a lifetime, how do you follow that? Pennell says he has the luxury of taking his time, and picking the best opportunity. He has already turned down a number of approaches.

"At this point I have no plans. I have some thoughts but not plans. I'm in no rush. I want to do some sailing when I leave. Then when the right thing comes along I'll be in a position to take it," he says.

"I won't try to follow this. I'll probably do something slightly different. You'll have to wait and see – sorry to be a tease. I genuinely don’t know yet. You can be certain it needs to be quite challenging, otherwise I'm not going to be interested. In many ways, running a project in this context is no different from in a corporate context – until you get to the Games itself, when it is entirely and completely different."

But the one thing Pennell does know is exactly where he plans to be at the moment his Olympics CIO experience finally comes to an end. "My official end date is 23 November. I get made redundant on that day. My plan is to be made redundant while sitting on Copacabana beach in Rio."