Case study: How eBay uses its own OpenStack private cloud
eBay runs one of its most crucial workloads – its website – on an OpenStack private cloud platform developed by itself
Online marketplace eBay runs one of its most crucial workloads – its website – on an OpenStack private cloud platform developed by itself.
Three years ago, eBay’s website was operated fully in its on-premises datacentre infrastructure.
“Today, 95% of eBay marketplace traffic is powered by our OpenStack cloud. It was zero three years ago,” said Suneet Nandwani, head of cloud engineering at eBay, during the sixth annual Cloud World Forum conference in London.
It started out as a developer cloud project in 2012, but today, eBay’s OpenStack-based private cloud is a multi-tenant, multi-region, self-service cloud that hosts the company’s customer-facing business-critical applications and the platforms its developers use. It has all the principles of a public cloud, but is in-house, Nandwani said.
The company wanted to build a robust, scalable, agile infrastructure that can support the IT demands of all its brands, including the PayPal and eBay websites.
Building a cloud infrastructure
Deciding it needed a cloud infrastructure, the IT team first developed its own private cloud facility, not based on OpenStack. But the eBay team soon realised that was "not such a sensible idea", said Nandwani.
It then selected OpenStack application programming interfaces (APIs) to build the private cloud facility because it was a mature, open-source system. “We wanted to avoid any kind of supplier lock-in. We were very clear about that,” he said.
eBay X.commerce
eBay's X.commerce is an e-commerce platform used by developers and businesses.
It incorporates a range of commerce-related applications, from open-source e-commerce platform Magento to online payment service PayPal.
Developers use the ecosystem to sell their own applications and build plugins to existing services such as Magento or Salesforce.com.
This platform is built on the company’s OpenStack cloud architecture.
Strict compliance and regulatory requirements for operating an online marketplace meant the IT team had to ensure its cloud infrastructure was secure and PCI-compliant.
Today, its cloud infrastructure runs as many as 7,000 server instances and the internet marketplace is yielding the benefits of running mission-critical workloads in the cloud, said Nandwani.
“For one, we have brought down the app-provisioning time for our developers from four weeks in the datacentre infrastructure to 30 minutes on the cloud,” he said. “We've also gained operational efficiencies and have saved double-digit million dollars on hardware expenditure.”
But that is not all. Cloud infrastructure has also provided other benefits, such as self-service flexibility, ease of software and application development and an automated infrastructure – all a distant dream until 2012.
“We have also brought down rack onboarding time drastically from 45 days to just three days,” Nandwani added.
Improved capacity management
One of the IT team’s main requirements was to have transparency of assets and features around capacity management. “We did not know what assets we had, which ones were running efficiently and how we could manage them. Cloud has enabled us to have asset transparency.
“We also needed sophisticated capacity management capabilities because we cannot afford to have the eBay website down at all,” Nandwani said.
But in its traditional IT, predicting capacity was tricky. “We would overestimate and overpay for capacity as lacking capacity is not an alternative for us,” he said. Cloud computing has helped eBay overcome this problem and the IT team is able to predict capacity requirements more accurately now.
The IT team at eBay has also been able to have showback and chargeback capabilities on the cloud.
IT chargeback is an accounting strategy that applies the costs of IT services, hardware or software to the business unit in which they are used. The feature is aimed to shift responsibility to users and encourage them to treat IT services as they would any other utility, unlike traditional IT accounting models where the centralised department would bear the IT cost. Reporting systems that leverage IT chargeback can help administrators to clearly see what factors are driving costs and to budget accordingly.
“By 2015, we hope to have full chargeback capability on our cloud,” Nandwani said.
Open-source challenges
But the move wasn’t without its challenges. “The trouble with OpenStack is that it is open source and we don’t have a vendor to go to who will fix our problems,” Nandwani said. “Our cloud problems are ours to deal.”
But the team pressed ahead with OpenStack because of the open standards and because the cloud engineering team had experience on the system. “We have developers who are experienced in OpenStack cloud platform and we wanted to develop a platform and share the code with the community,” he said.
If you are implementing open cloud with an IT team that has no experience on it, the process may be very difficult
Suneet Nandwani, eBay
Nandwani also cited a blog post by an eBay colleague titled, OpenStack is not cloud. “It raised a lot of eyebrows. But when you really look at it, cloud is a service while Openstack APIs are open-source technologies. To make these APIs a cloud service, the IT team has to add network design, configuration management, autoscaling capabilities, metering and chargeback features, high availability, log processing, monitoring and alerting, SLAs, capacity planning and customer support tools, among other things.
“All this is hard work and needs experienced professionals,” he said. “So, if you are implementing open cloud with an IT team that has no experience on it, the process may be very difficult,” he warned the enterprise customers present at Cloud World Forum.
To make applications, virtual machines and instances more productive and interoperable on the cloud, eBay’s IT team is also gradually increasing the footprint of Linux-based hypervisors and open-source systems.
“We are slowly shrinking our Microsoft and VMware estate,” said Nandwani. “VMware still forms a major part of our on-premises estate, but we hope to cut it down drastically in the next two to three years to have more KVMs [kernel-based virtual machines].”
Nandwani and his team are now looking to move more systems to its private cloud infrastructure as their long-term strategy. In the short term, his team is looking to optimise the cloud. “We have challenges around under-utilisation and are looking to resolve them,” he concluded.
