Jakub Jirsák - stock.adobe.com

News

How one red team exercise averted a new SolarWinds-style attack

Palo Alto Networks shares details of how its red teamers found and sealed a customer vulnerability that could have led to another SolarWinds-style supply chain attack

Alex Scroxton
By
Published: 28 Sep 2021 16:00

A competent threat actor can easily infiltrate an organisation’s development environment and carry out a SolarWinds or Kaseya-style supply chain attack in a matter of days, according to new intelligence from Palo Alto Networks, which is warning that many software companies have been lulled into a false sense of supply chain security in the cloud.

In a newly published report on the state of the cloud security market, which can be downloaded in full here, the firm’s Unit 42 team shared the story of how they successfully penetrated a major software-as-a-service (SaaS) vendor and ‘executed’ a supply chain attack in barely 72 hours.

The software company, which cannot be named for obvious reasons, engaged Unit 42’s services to break into its cloud software development environment having been spooked by the SolarWinds incident.

The Unit 42 red team took advantage of misconfigurations in the development environment to take control of the customer’s software development process in a similar fashion to how a Russia-linked advanced persistent threat (APT) group was able to penetrate SolarWinds’ platform to introduce a tainted code update.

“Risks related to supply chains have received a lot of attention in the media recently, what the discussions often overlook is that attackers don’t necessarily modify source code repositories to facilitate these breaches. They don’t have to. They find weaknesses in the software development pipeline and attack those,” wrote Matthew Chiodi, Palo Alto’s chief cloud security officer in the report’s preamble.

The red teamers posed as malicious developers with limited access to the customer’s continuous integration (CI) environment from where they tried to gain admin rights to the wider cloud environment – although this is somewhat different to what happened at SolarWinds it does similarly demonstrate how an attacker or malicious insider could take advantage of a CI repository.

To start, the researchers were assigned a DevOps role that the customer would commonly give to all developers in its environment, including access to its internal GitLab repositories. This was the customer’s first failing – had they followed recommended best practice around separation of duties for each developer, the exercise may not have worked so well.

Cloud resources

From here, the red teamers were able to download every GitLab repository from the customer’s cloud software storage location. Here they found approximately 80,000 individual cloud resources, including virtual machines (VMs), databases, storage instances, network infrastructure and network gateways within 154 unique repositories. Crucially, they also found 26 hardcoded identity and access management (IAM) key pairs, five of which were session tokens, the rest access keys.

From here, the red team escalated their privileges using the stolen IAM keys to exploit a misconfiguration in a specific Amazon Web Services (AWS) functionality, iam:PassRole, which enabled them to create an EC2 instance with admin access to the wider cloud environment.

Such access established, the red team had the full ability to perform any number of actions within the lucky customer’s, or unfortunate victim’s, cloud environment. This included poisoning CI ops to insert malicious code into their software development pipeline, although as this was outside the scope of the engagement, the exercise was stopped at that point.

Broader analysis of the customer’s security operations and response was then undertaken, which ultimately caused the suspicious activity to be surfaced at the customer’s security operations centre (SOC) by Amazon’s GuardDuty service, which worked as intended but was not properly configured across all the customer’s AWS accounts and therefore only spotted a small fraction of the overall exercise.

Nevertheless, once alerted, the SOC team was quickly able to identify each compromised IAM key used by the red team, and effectively deployed their security orchestration, automation and response (SOAR) tools to deactivate the keys and shut off access.

Security plan

Unit 42 has subsequently worked extensively with the organisation confirmed to implement a “shift left” security plan, elements of which (detailed in full in the report) include tighter, role-based access controls for developers to stop DevOps repositories going astray; the implementation of cloud platform detection rules for sensitive API requests coming from outside the organisation’s network range; and cloud platform detection rules for user-specific API requests directed to IAM service accounts.

“Red team exercises … demonstrate an example of how poor security hygiene in the supply chain can impact cloud infrastructure. The customer, in this case, a large SaaS provider, maintains what most would consider a mature cloud security posture. However, Unit 42 researchers found that 21% of the security scans they ran against the customer’s development environment resulted in misconfigurations or vulnerabilities, a number that squarely lines up with the industry average of 20%,” the Unit 42 team said in the report.

“Researchers believe it is highly likely that the techniques employed during the red team exercise could be successfully performed against many organisations developing applications in the cloud,” they said, although again it is also important to note that this particular attack path was not identical to the one used against SolarWinds.

The team said that despite much talk in the security community about the so-called shift left model, organisations are evidently continuing to neglect DevOps security, in part, they believe, due to lack of attention to supply chain threats.

Because cloud native apps have such a long chain of dependencies – such as open-source tools and various components from other vendors and developers, and these dependencies probably have dependencies too – it is critical for both DevOps and security teams to gain visibility into the full “bill of materials” in each cloud workload.

Too many, they said, still seem to believe that code scanning at the end of the development lifecycle is good enough, and as long as this mistaken belief persists, cloud development environments will continue to be targeted by threat actors.

“This was certainly the case for the SolarWinds breach as well as what could have been a major breach for our customer had the Unit 42 team not identified the customer’s supply chain vulnerabilities before a malicious attacker did,” they concluded.

Read more on Cloud security