BBC takes iPlayer video production to AWS cloud to beat bottleneck and storage woes
Having run Video Factory on AWS for over a year, BBC's Robert Shield explains how he overcame iPlayer bottleneck and availability challenges
For a video-on-demand service as popular as the BBC's iPlayer – it served three billion programme requests in 2013 alone – resilience, continuous availability and scalable storage are a must. And this means the back-end technology that handles the live media-processing must be competent.
The BBC developed an in-house tool – Video Factory - for producing the iPlayer's video content. Having run Video Factory on AWS’s cloud services for over a year, the broadcaster related how it overcame bottleneck, storage and availability challenges.
Speaking at AWS's Enterprise Summit in London on 21 October 2014, BBC principal software developer Robert Shield said: “The iPlayer receives about 10 million requests every day and we add 500 pieces of unique content to it daily.”
The primary goal for Video Factory was to have an in-house workflow, to replace the previous system that produced video for BBC iPlayer. “We simply could not fail in that, or the BBC iPlayer would stop working,” Shield said.
Prior to using cloud services, the iPlayer service faced the challenges of using a video-processing tool based on a traditional capital expenditure (capex) model and having to use a monolithic codebase, he said. It was not flexible or scalable, it needed a dedicated operations team and the storage was insufficient and unreliable.
Building resilience, scalability, efficiency and reliability
“There was another limitation – we could only have fewer updates and releases every year. This meant each release would have too many changes and it was becoming difficult to manage,” Shield said.
The BBC wanted a better workflow that would be more resilient, scalable, efficient and reliable.
The Video Factory architecture was built using AWS cloud services, Amazon Simple Queuing Service (SQS) and Amazon Simple Notification Service (SNS). The team also used AWS’s Elastic Transcoder – a cost-effective and scalable cloud-based media transcoding service.
The Video Factory team captures live broadcasts and sends them to AWS's cloud-based Simple Storage Service (S3) to await a transcode job. The content is sent to the transcoder and, once transcoded, the media either goes back to the cloud storage to be packaged on demand, when a request from the internet comes in; or channeled through the BBC’s other distribution services.
The whole system is automated in the cloud. Computer Weekly previously reported on how and why the BBC used AWS public cloud services to process media for iPlayer – but now, one year on, the BBC is exploiting more advantages of using the cloud. “Now Video Factory has well-defined boundaries on AWS EC2, it is scalable and stateless, and it is easy to make new components live quickly,” Shield said.
Shield told delegates at AWS's Enterprise Summit the cloud service had transformed the previously monolithic service, using 25 to 30 smaller components to undertake individual jobs. “In addition, instead of one heavy release a year, we can do about 35 smaller releases, spread across the year," he said.
Overcoming bottleneck challenges with cloud
“The new architecture has given us a continuous delivery model and made the iPlayer more resilient and flexible.” The team has also overcome storage bottlenecks. The iPlayer service gets about 6TB of broadcast content for storage every day and the team stores 24TB of data everyday on S3.
“The key to our success was the choice to use cloud services and the flexibility of continuous improvement and development,” Kiran Patel, senior product manager for Video Factory and Video Services for BBC Future Media wrote in a blog post.
"Using elastic computing and doing our video transcoding in the cloud meant the BBC had none of the bottlenecks that affected the old system.
“At 7pm, when all the national and regional news bulletins finish on BBC One, we have 18 live programmes ending at the same time. We can increase the size of our transcode farm at this time, to make sure all 18 are delivered without delay. Then we can reduce it after the peak, so we are not paying for resources we do not use,” Patel wrote in a blog post.
Cloud removes limitations and cuts costs
Prior to using Video Factory on AWS, the BBC's IT department could make live programmes available on the iPlayer in about three hours. Since July 2014 – with the advent of full automation and live-to-video on-demand processing in the cloud – programmes are available to view in three minutes.
“Without a cloud-based system, we would either have programmes waiting in a queue and delay availability, or we’d have to buy more hardware to cope with the peak – and have it sitting idle for most of the time,” Patel wrote.
Cloud’s elastic computing and storage model allowed the BBC to remove all limits in content production for the iPlayer. Previously, content volume was limited by capacity and contractual limits built into the old system. “We have doubled the total number of hours of video available on BBC iPlayer and increased the number of HD hours by almost 700%,” said Patel.
Cloud storage also allowed the team to make the BBC's content available for longer, for 30 days.
After running the live processing tool in the cloud for a year, the BBC's IT team is adding management features and increasing automation. “We can add management capabilities all the time,” Shield said.
"Now the Video Factory is more mature and the data from play-out is trusted, we can fully automate the process for programmes that have the accurate data," Patel explained. Automation has reduced manual effort and transcode costs, and improved the speed at which content can be delivered.
Live streaming improvements with cloud and Audio Factory
After building its confidence in the cloud with Video Factory production for on-demand services on the iPlayer, the IT team is now looking to improve the iPlayer’s live video-streaming service. “The BBC aims to deliver broadcast-quality content over IP using cloud services,” AWS’s Iain Gavin told the AWS Enterprise Summit.
“Building on what we have already done for Video Factory, and benefiting from the resilience and making all BBC channels available, we are refreshing the entire live video-streaming workflow to provide better quality streams, HD streams for the HD BBC channels and streams of all the national and regional versions of BBC One and BBC Two,” Patel said.
The live streams in BBC iPlayer and across BBC Online will be migrated to the new services before the end of 2014.
The team is now also looking to develop what it calls its Audio Factory, based on the same cloud principles.
“Video Factory has already improved the delivery of media for BBC iPlayer and all BBC Online products. The Media Services engineering team has embraced new technology and new ways of working to deliver cutting-edge and industry-leading software,” Patel said.
