Auto-tech series: Sysdig - Automation for developers, move (securely) faster

Passwords for robots

One of the challenges when using automated systems in the cloud is giving all the different components the access they need. Every step of the process usually needs some form of user credential, such as access to a GitHub repository, to the serverless functions used to trigger additional processes, or to the compute instances that run the application.

Each system may have a different method of storing credentials and performing authentication.

To make this work, you have to implement and manage your secrets effectively. For example, containers are commonly used because they make it easy to deploy and scale immutable software packages. However, one of the most common ways to pass a container dynamic information, like credentials, is through environment variables. If an attacker can get read access to the orchestration system (i.e. Kubernetes) they can also see and use those credentials.

Including credentials and API keys inside the containers themselves is a common mistake developers make when they try to automate these processes.

It is trivial to unpack a container and retrieve the keys.

The problem isn’t limited to containers – sites like GitHub and Bitbucket are often a valuable source of information for attackers because of how often credentials are left there. Of course, attackers also use automation themselves to harvest stray credentials quickly and efficiently, giving themselves more opportunities.

Talk to each other

Sometimes, credentials can be lying around in places you wouldn’t at all expect. Infrastructure-as-code (IaC) provides a machine-readable approach to defining and deploying automated configuration at scale. IaC includes many automation tasks that repeatedly recreate the same infrastructure and verify that the configuration is correct and secure.

During a recent security incident at a company that we work with, we discovered an attacker called SCARLETEEL that took advantage of how the IaC machinery works in practice in order to expand their attack.

In this case, the developers at the company used Terraform, the most popular IaC solution and stored a state file in an S3 bucket. This file contained all the necessary access management credentials needed … but Terraform also stores everything in plaintext. For automation to work, these tools need easy access to credentials that allow them to perform their tasks. The credentials must be kept somewhere. But because the technology is still pretty new, not everyone knows all of its nuances.

In this case, the problem was caused by a combination of issues. Ideally, the attacker should have not been able to get in in the first place, but they were able to exploit the deployment because the administrators didn’t realize the statefile existed and the S3 bucket also had excessive permissions in place. This allowed the attacker to read the file when they gained entry and they used these credentials to grant them enough access to steal some proprietary data.

Configuration mistakes

This scenario falls squarely into the ‘configuration mistakes’ pile, but it also provides proof that automation can be far more complicated than simply aligning your settings with software development best practices.

In order to make the most of automation in software development, we have to think about all the teams that might be involved over time. For example, infrastructure teams need to know about how developer tools like Terraform behave and security teams then need to know how to prevent or mitigate bad practices that might come up.

By working together these teams can ensure that automation works to everyone’s benefit.

Unfortunately, attackers who target cloud are discovering all of these tricks pretty quickly because, in a weird way, that’s their job. The best bet for developers and defenders is to join forces and learn from each other’s mistakes to keep up.