Datacentre lessons learnt from Heartbleed bug

Application automation, datacentre orchestration and access management

One important lesson datacentre professionals could learn from the Heartbleed bug incident is to enable application automation in datacentres.

Application automation offers a better response to security breaches across servers, the Gartner expert said. This is because a datacentre is home to thousands of web servers and updating the servers with automation will be easier and quicker.

“Having a good privilege access management strategy and datacentre orchestration are other ways datacentre professionals can respond better to such crises,” Heidt added.

Such an unprecedented security breach requires holistic action. IT professionals must have good relationships with technical experts inside and outside the company to solve the problem, he further advised.   

Companies that had provisioned for datacentre orchestration and centralised server management, as well as having up-to-date management tools, were able to respond quickly to the Heartbleed bug crisis.

Datacentre disaster recovery strategy

While at a technical level Heartbleed had fewer lessons, it offered lessons on how datacentre owners should react when the news broke, some experts have said.

Companies that had provisioned for datacentre orchestration and centralised server management, as well as having up-to-date management tools, were able to respond quickly to the Heartbleed bug crisis

Another important lesson for datacentre managers is that open source hardware isn’t necessarily risk-free.

“Any datacentre operator should have been able to provide cool, calm advice to its customers, and should have had the tools in place to rapidly and effectively patch OpenSSL to get rid of the problem – and then advise customers to change their passwords,” said datacentre expert and Quocirca director Clive Longbottom.

"There was far too much FUD [fear, uncertainty and doubt] around this – too much ‘advice’ to change all passwords now –  which only makes the problem worse, as the changed password could be compromised,” he added.

Server virtualisation provider VMware, which has nearly 500,000 customers, started issuing Heartbleed patches this week. As many as 27 VMware products were affected by Heartbleed. 

“Throughout the week commencing 14 April, VMware will be releasing product updates that address the OpenSSL Heartbleed issue. VMware expects to have updated products and patches for all affected products by 19 April,” its security announcement email to users read. 

But some VMware users took to Twitter to moan about the provider’s security patches – that the update was slow and came late.

Each operator should have been able to rapidly evaluate the scale of the issue and advise accordingly, experts said.

Such a datacentre disaster recovery strategy and processes should have already been in place and datacentre professionals must only be scaling that up to respond to the Heartbleed incident, not modifying it or devising a new strategy after the incident, added Heidt.

Ethical hacking tests

A well-run professional datacentre should have consultancy services available to help its customers test their systems in advance, and it should implement training for staff to make them aware of information security threats, according to London-based datacentre provider City Lifeline.

An example is “penetration testing”, otherwise known as “ethical hacking”, where a benign expert attempts to evade the security precautions taken by the target company and gain access to confidential information. The expert reports back to the company on its success, with recommendations for improvements, said Roger Keenan, City Lifeline’s managing director.

“Although on this occasion the process would not have identified Heartbleed, it provides datacentre users with confidence that it has identified and mitigated against many other, more common, more well-known threats,” he said.

Managing customer service expectations amid the crisis

For datacentre operators, how they manage customer services and how they deal with the OpenSSL vulnerability appropriately are the big issues.

“If an operator was affected and believed customers’ passwords had been put at risk, they have to clearly state that they will fix the problem and the appropriate time users must change passwords,” said Andrew Kellett, principal analyst, infrastructure and software, at Ovum. Such communication was not very clear this time around, he said.

“Some operators and big tech giants reassured customers, saying they are not at risk, but it was not clear whether there was a breach and it was fixed or whether their servers were not affected at all,” said Kellett.

“A holding page on their website could explain what it means to customers and what steps the operator is taking,” added Longbottom. If the operator is dealing with highly sensitive data, then it should suspend logins and deal with each customer separately, experts advised.

There will always be another Heartbleed, and it is likely that the Googles and Amazons will handle the problem very efficiently. It is the smaller, medium-sized datacentre providers that may take time to respond, Kellett said.

His advice to CIOs: “Look towards your service level agreements and see what it says on security and check with datacentre providers that they have, if they had the problem, dealt with it.”