Xen Orchestra latest victim of Salt cryptojackers

Xen Orchestra, a web-based management service for users of Xen hypervisors, has become the latest known victim of cyber criminals exploiting two critical vulnerabilities in SaltStack’s open source Salt remote task and configuration framework, which has left thousands of cloud servers all over the world exposed.

First disclosed last week by F-Secure researchers, the Salt vulnerabilities are highly dangerous and easy to exploit. Left unpatched, they give malicious actors the ability to execute code remotely with root privileges on Salt master repositories to do any number of things.

In the case of Xen Orchestra, its attackers exploited the vulnerability to run a cryptominer on a number of its virtual machines (VMs), which the firm spotted early on 3 May when several services on its infrastructure became unreachable.

Further investigation uncovered a rogue Salt minion process mining coins – in common with blogging platform Ghost, which was hit by a similar attack – which caused a notable spike in central processing unit (CPU) usage.

“A coin mining script ran on some of our VMs, and we were lucky nothing bad happened to us – no RPMs affected and no evidence that private customer data, passwords or other information has been compromised. GPG signing keys were not on any affected VMs. We don’t store any credit card information nor plain text credentials. Lesson learned,” Xen Orchestra’s founder Olivier Lambert wrote in a disclosure blog.

Lambert said it was clear the organisation had been caught up in a very broad cyber attack affecting a great many people, and said F-Secure’s assessment that there were 6,000 vulnerable Salt instances detected in the wild was something of an understatement.

He acknowledged that Xen Orchestra had got off lightly, being hit with a very generic payload that was probably built very quickly to make some quick cash off the vulnerabilities.

Xen Orchestra was left exposed because it had trusted SaltStack’s communication protocols to flexibly add new VMs without using a virtual private network (VPN) or secure tunnels, which was a mistake because it let the attackers access the available Salt master port to inject their payload. It had also not yet seen a patch in its master distro repository.

It has now discontinued its use of SaltStack for the short term, and removed it from its VMs, as well as changing all its system passwords, tokens and keys with external services, and enhancing its network privacy.

“Luckily, the initial attack payload was really dumb and not dangerous. If you are running SaltStack in your own infrastructure, please be very careful. Newer payloads could be far more dangerous”

“In short, we were caught in a storm affecting a lot of people. We all have something in common: we underestimated the risk of having the Salt master accessible from outside,” said Lambert.

“Luckily, the initial attack payload was really dumb and not dangerous. We are aware it might have been far more dangerous and we take it seriously as a big warning. The malware world is evolving really fast: having an auto update for our management software wasn’t enough.

“If you are running SaltStack in your own infrastructure, please be very careful. Newer payloads could be far more dangerous,” he said.

More technical details of Xen Orchestra’s experience can be read on its website.

Alex Peay, senior vice-president of product and marketing at SaltStack, said it had taken immediate action to remediate the vulnerability, develop and issue patches, and communicate widely to customers about the affected versions.

“Although there was no initial evidence that the CVE had been exploited, we have confirmed that some vulnerable, unpatched systems have been accessed by unauthorised users since the release of the patches,” he said.

“We must reinforce how critical it is that all Salt users patch their systems and follow the guidance we have provided outlining steps for remediation and best practices for Salt environment security. It is equally important to upgrade to latest versions of the platform and register with support for future awareness of any possible issues and remediations,” said Peay.

Peay pointed out that F-Secure’s estimate of 6,000 vulnerable instances of exposed Salt masters represented a very small portion of its installed base. He said clients that had followed fundamental internet security guidelines and best practice were not affected.