What it takes to get DevSecOps right

At SP Digital, the digital arm of Singapore’s largest utilities supplier SP Group, embracing DevOps practices has enabled it to roll out new software releases much faster than before.

Take the SP Utilities mobile app that lets households manage their utilities consumption, for example. The firm now updates the app about once every fortnight, compared with every couple of months before its foray into DevOps about four years ago.

Colin Leong, vice-president of engineering at SP Digital, said when his team took over the app from an IT supplier at the time, software releases were sporadic, with definitions of what needed to be done upfront.

“We can now push out new features quickly, as well as address any kinds of bugs or defects that may crop up,” said Leong. “We have a much stronger pipeline, and I think the ability to push that out is much better than it was in the past.”

Just as SP Digital has improved code quality through DevOps, it is now looking to shore up security through DevSecOps, where security considerations are baked into the early phases of software development.

According to IDC, a technology research firm, DevSecOps will drive at least 50% of new applications in Asia-Pacific by 2024, fuelled by shorter software development lifecycles.

“Old security processes that put security at the middle or end of the process are just too expensive and inefficient now. Shifting security left – all the way to the planning stage – can dramatically improve efficiency and decrease cost”

“Old security processes that put security at the middle or end of the process are just too expensive and inefficient now,” said Gina Smith, research manager at IDC Asia.

“Shifting security left – all the way to the planning stage – can dramatically improve efficiency and decrease cost. The bottom line is that it jumpstarts the output of quality code, which is what it is all about,” she added.

Smith said as more enterprises rely on open source and cloud technologies, as well as application containerisation, they will face a “complicated set of challenges” which a mature DevSecOps policy will help to address.

“Building security planning, testing and monitoring into every phase of the DevOps pipeline is about bridging the age-old division – and enmity – among developers, IT and security,” she added.

SP Digital’s Leong said while his team is currently exploring tools to enable DevSecOps practices, a bigger challenge is that the firm’s security capabilities are heavily centred around enterprise security.  

So, it recently hired an application security specialist who has been helping to shape SP Digital’s DevOps practices and also get the tools in place to build security into its development pipeline, he said.

Nigel Kersten, Puppet’s field chief technology officer who was part of the famed site reliability engineering group at Google, stressed the importance of deploying automation at scale in DevSecOps practices.

“There are a few common errors we see that enterprises are facing – the biggest one is trying to implement DevSecOps without scaled automation that is well understood and trusted by all the relevant stakeholders.

“Building security planning, testing and monitoring into every phase of the DevOps pipeline is about bridging the age-old division – and enmity – among developers, IT and security”

“Without that, organisations will end up with the same manual processes and the same conflicting incentives. Then, instead of DevSecOps, these businesses are left with just Dev, Sec and Ops,” said Kersten.

Organisations, however, will have to pick tools that developers want to use. “Enterprises cannot just force a security or an infrastructure tool on developers – it needs to have an interface that fits, is usable and could be programmatically driven through application programming interfaces,” Kersten added.

There’s also change management to consider. Kersten said organisations will have to do the difficult work of getting multiple teams with different incentives to work together and make change management happen. This change is hard and there are no easy answers.

“What we do see repeatedly is that the companies who succeed at collaboration between development and operations via scalable automated solutions are the same ones who succeed at doing the same with security,” said Kersten.

Sam Hunt, vice-president of GitHub in the Asia-Pacific region, said another challenge with DevSecOps is managing false positives.

“Embracing DevSecOps processes will inevitably increase the rate of vulnerabilities being discovered. As such, false positives are bound to happen, which erode developer confidence in the value of security checks. How teams handle these will make or break the DevSecOps culture.

“Teams need to prioritise bugs in terms of importance and impact, to determine how they should fix them. By operating in a security-first workflow, teams can identify the bugs that have the most critical impact and take steps to manage over time,” said Hunt.

Puppet’s Kersten noted that as DevSecOps is fundamentally about recognising that security can no longer be a siloed function, it can become the base structure of a cyber security strategy.

“Ideally, companies have operations teams employing a high degree of automation via self-service interfaces, with developers using agile methodologies. To achieve that, the most effective approach is to enable and amplify collaboration throughout the software delivery lifecycle, from design to deployment and beyond,” he said.

That requires every person who is part of that lifecycle to be security aware, with developers coding with security in mind, said Vishal Ghariwala, Red Hat’s regional product management director for application platforms in Asia-Pacific, adding that some developers may need training, as security has not always been a focus in traditional application development.

Ghariwala suggested putting in place a framework to help an enterprise determine its security requirements, risk tolerance and to conduct a risk-benefit analysis. “For example, what amount of security controls are necessary for an app? How important is speed to market for different apps?”

GitHub’s Hunt said although other traditional security responsibilities, such as infrastructure security and identity management, are not as affected by DevSecOps, as enterprises shift to “infrastructure as code”, “policy as code”, or other “as code” models, DevSecOps processes will help to automate reviews in other areas of security.

Meanwhile, DevOps remains a work in progress at SP Digital. Leong’s team has been organising learning exchanges between IT teams, including infrastructure, operations, application development and security, to share best practices.

“We’ve had some successes, and we’ve also helped the teams with some level of automation that has improved the effectiveness of IT projects,” Leong said. “We aspire to hit a level where there’s a lot more self-service and that their concerns can be taken care of in a much more automated way.”