Vulnerable firmware in enterprise server supply chain

Two serious vulnerabilities in baseboard management controller (BMC) firmware used by Lenovo, Gigabyte and other manufactures in some server products can be exploited to hide malware from the operating system, hypervisor and antivirus systems, researchers warn.

A BMC is a specialised service processor that monitors the physical state of a computer, network server or other hardware device using sensors and communicating with the system administrator through an independent connection.

The vulnerabilities were discovered by researchers at security firm Eclypsium during an investigation into BMC firmware vulnerabilities in the supply chain.

The vulnerabilities in third-party BMC firmware from Vertiv (formerly Avocent) made enterprises susceptible to data loss and permanent damage to hardware, while enabling attackers to persist even across new operating system installation. 

The first vulnerability is a failure in the update process to perform cryptographic signature verification before accepting updates, while the second relates to command injection vulnerability in the code in the BMC that performs the firmware update process.

Both of these issues allow an attacker running with administrative privileges on the host (such as through exploitation of a different host-based vulnerability) to run arbitrary code within the BMC, and malicious modifications to the BMC firmware can be used by an attacker to maintain persistence in the system and survive common incident response steps such as reinstallation of the operating system, the researchers found.

An attack could also modify the environment within the BMC to prevent any further firmware updates through software mechanisms, thus enabling an attacker to disable the BMC permanently, and the update mechanism could be exploited remotely if the attacker has been able to capture the administration password for the BMC, the researchers said.

Lenovo was the first to go public with the discovery, stating in a security advisory that the company had become aware that in certain legacy Lenovo ThinkServer-branded servers, a command injection vulnerability exists in the BMC firmware download command.

“This allows a privileged user to download and execute arbitrary code inside the BMC. This can only be exploited by authorised privileged users,” the advisory said.

Lenovo thanked the Eclypsium research team for notifying the company of the vulnerabilities and urged customers to ensure they updated the BMC firmware on affected products. Lenovo also advised server customers to restrict authorised privileged access to trusted administrators.

GigaByte published an updated version of the firmware to fix the command injection vulnerability for systems using the AST2500 on 8 May 2019, but has not released an advisory for this issue. The AST2400 firmware version remained unpatched as of 21 June 2019, the researchers said, adding that Vertiv has not responded to their communications.

In addition to Lenovo and Gigabyte products using BMC firmware from Vertiv, the Eclypsium researchers said that Gigabyte also provides motherboards to smaller system integrators that then build complete systems under their own branding. 

This means the vulnerable firmware was included in servers from a variety of suppliers, including Acer, AMAX, Bigtera, Ciara, Penguin Computing, and sysGen.

The wide spread of suppliers affected highlights an important challenge for the industry, the researchers said. “Most hardware vendors do not write their own firmware and instead rely on their supply chain partners. Firmware is quite commonly licensed from a third party and used with little modification, allowing vulnerabilities to extend to many different brands and products,” they said in a blog post.

In light of this fact, the researchers said manufacturers should test thoroughly any firmware they license for vulnerabilities, and enterprise security teams should perform security scans of device firmware as part of accepting any new piece of hardware.

The researchers also note that the scope of BMC vulnerabilities extends far beyond the newly discovered  pair of vulnerabilities, and is not limited to just a few suppliers.

“Industry stalwarts HP Enterprise and Dell have both been found to have serious firmware BMC vulnerabilities of their own. As our previous research into Supermicro demonstrates, vulnerabilities in server firmware are common and may have a significant impact on enterprise IT Infrastructure. They allow an attacker to persist undetected inside a server or even permanently disable the victim server,” the researchers said.  

As attackers and nation-states target higher-value assets, the researchers aid BMC and other firmware inside critical servers provide a particularly strategic target, as they can be used to “brick” the server and its contents permanently.

A spokesman for Vertiv said: "As a leading provider of BMC firmware to the OEM community, Avocent began working with key customers as early as 2012, before it was common in the industry, to encrypt and provide verification that the software or firmware being updated was from a trusted source. In 2014, Avocent released a feature upgrade for the MergePoint EMS BMC firmware platform that included verification signing. During the past year, we were alerted to the command line concern and quickly developed and released a patch for our customers.

“We are not aware of any issues related to this, and it’s important to note that the issue identified by the researcher could not have been used to penetrate a network or system. Only someone with access to the system could exploit it. We appreciate researchers bringing matters like this to our attention. It helps strengthen our products and provides an opportunity to remind all consumers and businesses to regularly install software updates and patches to keep their systems current.”