Best practices to beat container misconfiguration

Good strategy means less is missed

Obviously, managing network configuration and containers can be tricky – from figuring out resource allocations for container scaling, or how to maintain state and persistence while avoiding drift, on top of paying attention to securing images, private and public registries, permissions and the like.

For Firment, Kubernetes and security issues can be summarised as over-privileging, over-provisioning and over-exposure. For example, of multiple generated permissions, many will remain unused. “And people are still using root or superuser access to run their containers,” Firment notes.

Francesco Beltramini, head of technical solutions at consultancy ControlPlane.io, recommends IT decision-makers ensure they are taking in what is happening across the organisation, not just in IT.

He says that chief information security officers (CISOs) and other business leaders may need to map the IT more completely to the organisation’s existential risk, working across cyber security disciplines, and filling gaps between technical teams and decision-makers.

“Assurance spans ongoing development, testing, public and private datacentres, operations and security,” he says. “Ask what you’re building and about outcomes, then ask what’s the worst that could happen, and then ask how to reduce the risk of bad things happening.”

This assurance work needs to be repeated with threat modelling and hardening to ensure security by design and default.

For instance, to ensure Kubernetes is secure, IT leaders should put in place careful cluster administration at all levels of potential attack trees and threat surfaces associated, given the increasingly layered and complicated tech stack arising from containerisation.

An increasingly complex stack is also harder to understand, which, as Beltramini points out, widens the gap between technical teams and the technology adoption roadmap.

Beltramini recommends IT leaders cascade choices down from the choice of operating model – developer self-service in infrastructure as code modules or service catalogues, or via a central platform team – and structure the organisation accordingly. Security and development teams should also collaborate closely to reduce delays to the system. In addition, acceptance criteria should be defined early and regulatory non-compliance blocks to go-live of systems avoided.

Beltramini recommends IT leaders harness automation, encryption and native integrations, with all departments involved in authoring or changing shared code where possible. That means granular per process, container and pod policies, and detailed runtime configurations and resource accounting. Security profiles should be customised around system calls, file system access, binaries, libraries, capability restriction and so on.

A systematic approach is needed to cover all the bases with this degree of complexity in the stack. For instance, just in each pod alone, considerations span images, applications, operating systems, users, any baked-in secrets and more, says Beltramini.

Chris Jenkins, principal chief architect at Red Hat, also highlights thorough the need for consideration of containers and their orchestration, as well as what’s inside containers. Don’t forget that many security failures can come back to a neglect of basics such as updating and patching, not to mention ill-considered use of private versus public keys.

Keep it simple when you can

Jenkins urges IT leaders to start clean, keeping it simple where possible. “You want to start by doing something right. Give them a [clean] base image on which they can start, and minimise pain,” he says.

Jenkins urges software developers only to use packages needed to run the application. For instance, if 50,000 packages are installed that are never used, they are likely to expose a vast number of potential vulnerabilities. Instead, Jenkins’ approach is to start with a blank page on which to base the application development work, where it is possible to understand the security status or health index of a specific image.

“We produce base images every six weeks,” he says. “Then we run code analysis to check all the code. Developers will also have dependencies of code which will call in other libraries.”

The final immutable bill of materials or recipe should equal and be verified against this specific image. If anything changes, if a clarification goes wrong or the key doesn’t match, then don’t deploy it, he says.

Smaller organisations may obviously find this level of risk management challenging. It may be possible to look at splitting up all the tasks, for instance, by cycling through the different tasks on different days of the week, to get everything covered cyber security-wise.

A variety of tools exist to help with this, particularly in the open source arena.

Dinesh Majrekar, chief technology officer (CTO) at services provider Civo, says some tools can help organisations “shift-left” on container security. They offer functionality such as scanning containers during the build and reporting into the continuous integration and continuous delivery (CI/CD) pipelines on known issues. Examples include on whether network ports are open, how libraries are defined, or whether a version of Java or Node.js is out of date.

Giving developers continuous feedback is “really important”, according to Majrekar. He recommends opting for least privilege and permissions, minimal installations and dependencies – although IT decision-makers need to balance this against debugging requirements.

“You can make mistakes everywhere,” Majrekar adds. “Using Kubernetes, there are 100 different things you can change. You might not spot something you should have said ‘yes’ to which is currently set at ‘no’, and that isn’t necessarily the containers, but the managed Kubernetes environment you’re running within.

“You’ve got to now orchestrate changing the password and deploying your code into production at the same time, for example,” he says. “You need to use your experience as well in terms of secrets, storage and things like that instead. This comes back to a little bit of education, and I think specific security-focused training [such as certified Kubernetes security specialist, or CKS] and/or reading up.”

Regular penetration and configuration testing can be key to reveal role-based access vulnerabilities and missing best practices, such as around network segmentation or secrets management – not least because the potential issues are constantly changing as well, he says.

“Smaller companies might need to roll the dice more because of cost,” adds Civo chief executive Mark Boost. “You might sometimes just get things working, push them out, and before you know, it is out in production even though you didn’t get around to hardening it. So, the recommendation there is to just make sure you come back to it.”

Organisations should double-check things for themselves. Do not simply leave it all to a managed services provider, assuming they’ve taken on that responsibility and all the security is handled.

“Containerisation can be massively powerful, but with great power comes great responsibility,” adds Majrekar.

Reduce public attack surfaces

Crystal Morin, cyber security strategist at Sysdig, notes that “something like 66% of registries” are still public – not internal, not private, not supplier-managed. She says these are often “just pulling something off the internet, throwing it into your infrastructure”.

Scans are happening while updates are being pushed into the public-facing image, and the IT security team may be unaware it is happening. This is an issue Morin believes IT security and developer teams need to address.

“Turning away from public resources for container images is going to reduce supply chain risk and misconfigurations,” Morin adds.

“Shift left, shield right” remains best practice: ensure enhanced security to begin with and continue protection on the back end. Secure containers in production with real-time threat detection and policies for timely alerting and response, including automation of incident response.

She notes that without automation, time, capacity or capability can be in short supply.

“Cloud-native, security-focused organisations are realising now that security is an organisation’s problem. Everyone needs to be aware of what they do and how they can help,” Morin says.

Integrating security across the entire organisation also delivers value to all users and ultimately to every customer.

“If I go in and talk to others about security, they might understand maybe 40%,” says Morins. “We need to collaborate better across the business, coordinating together, not just as the technical nerdy guys behind the scenes. Shift left, make development more secure. And that’s obviously a much bigger picture than just containers.”