How web giants are making datacentres lean and green

Facebook: Engineering efficiency

Furlong outlined how Facebook went through a major exercise to improve datacentre efficiency around the turn of the decade, looking comprehensively at software, servers and datacentres. Its development of the HipHop for PHP engine to speed up the execution of code, and later the HipHop Virtual Machine (HHVM), were key developments. 

“We’d have five times as many servers today if we hadn’t done HipHop and HHVM,” he said. 

The company also optimised servers for efficiency, taking out unneeded components and making them easier to maintain. At the datacentre level, it switched to evaporative cooling and improved electrical systems. 

“Together, these measures gave us about 38% more efficiency and 24% less cost in our Prineville datacentre. We’ve subsequently replicated the approach in all our buildings,” said Furlong.

These improvements prompted Facebook to create the Open Compute Project (OCP) in 2011 to encourage other organisations to join its mission to engineer “the most efficient server, storage and datacentre hardware designs for scalable computing”. With around 60 members, the group looks at improving efficiency at all levels of the stack, and between those levels. Furlong admitted that the initiative had failed to attract many European members, but hoped that would change in future.

These measures gave us about 38% more efficiency and 24% less cost

Thomas Furlong, Facebook

Today, Facebook is aiming to improve efficiency even further by gaining a better understanding of its changing utilisation levels and workloads over time, as well as more granular control over its servers and clusters so it can keep utilisation levels high while still provisioning appropriately for workload peaks and user growth. To that end the company is developing a comprehensive datacentre infrastructure management (DCIM) solution. 

Furlong said the biggest challenge for its software engineers has been to uncover and clean up the disparate sources of data needed to make the solution work effectively. “Now we are constantly adjusting and tweaking clusters to gain full utilisation,” he said. While in 2012 that involved a lot of manual tweaking, “today we can do it very quickly using software”.

eBay: Doing as the business bids

While Furlong noted Facebook had been “very focused on the tactical” aspects of improving efficiency, he admitted it was also important to figure out the business implications. This is an area where eBay has excelled. 

“How do you explain to your C-suite that you need another $100m for the datacentre? They just see it as an infrastructure overhead. We decided to create a metric that we call digital service efficiency (DSE),” said Nelson in his conference presentation.

“Our CFO cares about the cost of putting the infrastructure together, our chief sustainability officer cares about environmental impact, and the CEO, COO and the rest care about the revenue generated from this engine,” he added.

"Everyone is looking at which knobs they can turn to make the numbers better."

Dean Nelson, eBay

In 2012, eBay processed 4.3 trillion transactions for its 144 million active users, where a transaction is defined as a single URL request (including the various back-end calls made by the system in response to the original user-generated request). “We realised everything done by a user is actually consuming kilowatt hours,” said Nelson. 

By dividing the total number of kilowatt hours consumed by the total number of transactions processed, eBay was able to come up with what is essentially a “miles per gallon” measurement of datacentre performance. This top-level metric allowed the company to develop specific second-level metrics that showed how it was performing each quarter in terms of total energy efficiency, carbon consumption and revenue – all measurements that show how datacentre efficiency is directly affecting business performance in terms relevant to the various concerns of its executives.

“This promotes continuous improvement, because everyone is looking at which knobs they can turn to make their numbers better. That means you can design datacentres that are really efficient, and operate them so they continue to be efficient,” said Nelson. 

And because these metrics are business-driven, they are technology agnostic. “It doesn’t matter what solutions are underneath. We let engineers do the engineering – we just give them the parameters,” he said.

Having used DSE to guide its decisions for 18 months, Nelson said the approach had been a resounding success. Among the efficiency solutions it has deployed are liquid cooling, modular “rack and roll” datacentres, high-density clusters, solar arrays, Bloom fuel cells and onsite power generation using natural gas, as well as a host of lower-level tweaks.  

“We’ve seen a 40% increase in transactions and added 34% more servers to our infrastructure, yet we have only consumed 24% more power. This means over that 18-month period we’ve avoided using 1.6 megawatts of power that we would have needed had we not turned the knobs we did,” said Nelson.