Molotov.tv streams 24/7 with Dell EMC and Scality object storage

244 servers in 11 racks

With half of its staff working on the technical side, Molotov is something of a digital powerhouse. Its datacentre comprises 244 servers in 11 racks dedicated to 24/7 transformation and distribution of programming in numerous formats.

Developers constantly refine their algorithms and have even automated procedures to maintain activity in case of an incident. Engineers watch for the slightest drop-off in performance, and with all the applications built as microservices across 240 virtual machines, load balancing takes place to account for relative demand for programming.

Video streams are relayed as close as possible to users by content delivery networks (CDNs), to cache servers that dedicated providers install in access providers’ datacentres.

These servers have to absorb customer demand during programme delivery, and also need to be able to provide access to the whole range of programming available.

“We retain all the programming we deliver so that viewers can restart them from the beginning, but also because we offer the ability to resume viewing later on. For that reason we need storage with a lot of capacity, which is scalable and with quick read times.”

Currently, Molotov’s stored data represents around 6PB and grows by 25% to 30% per annum.

“Fundamentally, we needed to put in place a way for hundreds of thousands of users to connect simultaneously,” says Ouicher.

This part of the service is dependent on the database that feeds the user interface that customers control from their terminals.

Plug-and-play servers from Dell EMC

In 2013, to help design the infrastructure that would allow it to underpin its service three years later, Molotov called on a specialist in the area, namely the Paris-based integrator Iguane Solutions.

Iguane was founded in 2000 by Benjamin Bejbaum, who went on to create the video-hosting site Dailymotion. Also among its achievements is Deezer, the French online music rival to Spotify.

The provider advised Molotov to base its infrastructure estate on Dell EMC hardware.

“Our idea was to limit the number of products, to simplify administration and to support growth in activity by the quick and simple addition of new nodes,” says Iguane chief operating officer (COO) Cédric Le Moan.

“Iguane had chosen Dell EMC because its servers are plug-and-play. There’s not even a need to use a screwdriver. It all just slides in on rails.

“Elsewhere, Dell EMC machines are equipped with iDRac controllers, which put all server information on a central console and from where you can take control of any node,” adds Le Moan.

The COO says the challenge for an important infrastructure such as that at Molotov is to not fall into the time-consuming trap of administration.

2U rack servers dedicated to stream encoding

For stream encoding, Molotov invested in Dell EMC PowerEdge R700s. Their advantage is that they fit into 2U of rackspace with enough room to add GPU cards that are needed to provide the processing power for video format conversion.

“The initial option was to use dedicated video transcoders, but that wasn’t a satisfactory solution,” says Ouicher. “The GPU cards gave us more flexibility in application development. Iguane helped us make models to evaluate the number of GPUs and in how many servers our applications needed to process what quantity of streams.”

Ouicher doesn’t want to disclose the details of the R700 estate deployed to date, saying that it constitutes a business secret.

User connections are handled by Dell EMC R400 servers. They’re largely the same as the R700s, but fit into 1U of rackspace and can’t have a GPU added.

The R400s execute applications connected to databases and it’s the amount of RAM that governs the number of virutal machines (VMs) and the connecting network that are important.

Object storage from Scality for minimal latency

The part of the infrastructure that Alexandre Ouicher pays most attention to is storage.

Molotov decided on Dell EMC disk shelves (JBODs) that hold 12 hard disk drives (HDDs), each of 14TB in capacity, connected to PowerEdge R640 server hardware. These twin-processor 1U nodes have two Fibre Channel cards and each is capable of controlling eight JBODs.

“This part of the infrastructure is particularly scalable. The advantage of the R640 is that it is simple to add RAM to, or even to change processors, so that we always have enough power to manage capacity that always grows,” says Ouicher.

“We add to the storage and other resources three times a year, knowing that we will never go beyond eight JBODs per server due to questions of security.”

On the software side is object storage, managed from the R640 servers by Scality Ring.

Scality is particularly adapted to Molotov’s workloads. It is a system that suffers least from latency where writes are more numerous than reads.

Effectively, Molotov has to have all streams from all channels ready to transmit, in all formats, even though subscribers don’t consume them in a systematic way.  Reads are uniquely those streams that must be sent via the CDN.

“I’d also add that Scality allows us to lose up to 30% of JBOD capacity without losing data,” says Ouicher. “To optimise our storage we don’t need data deduplication. Instead, we just have to regularly rebuild Scality’s index.”

Again, Ouicher refuses to be drawn on details, but he explains that to limit any explosion of data that results from users not cleaning up queued programmes on their watch lists, the Molotov IT team reorganises things, even to the point of re-encoding the oldest files.

Avoiding incidents

Molotov’s infrastructure is monitored in several ways. Youbora provides specialised video stream monitoring that’s capable of showing connections between datacentre activity and user experience.

Meanwhile, applications that are built as micro-services are monitored by Datadog. Then, all business and technical metrics, system logs included, are covered by ELK’s elastic search and analytics.

In the case of other technical snags, the technical team writes Python scripts to help avoid any issues. Molotov points to monitoring the expiry of SSL certificates as an example of this.

Thanks to such monitoring capability, the “natural” security of object storage and the quality of the infrastructure, Molotov hasn’t experienced an outage since 2016.

“The only notable problems have been on the internet side of the network,” says Ouicher. “But even that wasn’t catastrophic because everything went via the CDNs.”

According to Ouicher, Molotov’s infrastructure isn’t difficult to administer. Iguane Solutions carries out some of these tasks, such as regular additions to hardware resources at its datacentre.

Molotov, meanwhile, occupies itself with system administration operations. Among all this, the backup load is lightened with regard to the volume of data in production. And for good reason: only the databases connected to user activity and the programme catalogue are protected. The rest of the data stored – the videos – are already a backup copy.