CI/CD series - Veracode: up the code pipeline without going down the rabbit hole

Farrington writes…

Platforms such as GitLab, AWS Developer Tools or Azure DevOps are increasingly favoured by teams looking to optimise software development, unlocking faster velocity by leveraging the power of the cloud to perform software build and tests in parallel.

In the past, development teams needed to wait-in-line, for on-premise build servers to complete tasks before having work completed. Today, the economics of these online CI/CD platforms allow steps to take place across a distributed compute model, so that release pipelines can be compressed.

Code isolation situation

I want to make note of the fact that, out in the real world of live production code, different CI/CD release patterns can be extremely powerful for DevOps teams.

Isolated code can be controlled by feature flags and routing rules that limit how many users experience new code. The ‘canary release pattern‘ is a popular example technique, as it allows deployments to take place at any time of day, but the user experience is carefully controlled so that initially only a trickle of users are exposed to new functionality. The gradual increase of flow of users can be controlled by scripts which look for errors appearing in logs. Based on certain tests, the roll-out could be automatically paused or rolled-back, without engineers needing to be paged in the middle of the night. These tests might also include automated security scans, for example with Dynamic Analysis, which could conceivably ‘vote’ for a rollback if say, a SQL injection flaw is found on the main login page. Once the isolated code has passed a series of tests during the release pipeline and once in production, it can then be allowed to survive if negative test results are below a designated threshold.

People also often ask about CI/CD in terms of how development sits next to [continuous] deployment.

Today, very few organisations have reached the level of maturity to support (automated) continuous deployment. CI/CD are terms which provide teams with the ability to release ‘deployable’ code at will, subject to when the business wishes for that to happen. In the recent Puppet State of DevOps report (2019), 56% of those surveyed agreed that business reasons mean that they choose not to automatically deploy. Consider that only 7% of developers, according to the same study, are able to remediate critical security vulnerabilities in less than one hour.

Getting funky on functional

Developers will typically write unit tests to help validate the correctness in a function. These can range in complexity, but in essence, they should be deterministic. For example, it could be validating that a function correctly sums inputs properly, or perhaps that it sorts a range of data points in the expected way. Because of their nature, unit tests will usually run quickly and are a normal part of a build process, fitting well within a CI/CD pipeline.

Increasingly, developers are using tools to perform security unit tests on code changes. In comparison, functional tests will often seek to ‘exercise’ a running application by executing a script and grouping together a number of user actions so that the behaviour of the application can be validated. Functional tests can also be undertaken within a CI/CD pipeline, but will generally require more time to complete. Depending on the nature of the project, the development team may elect for some functional tests to be performed following deployment, whilst retaining the ability to roll-back the release if need be.

Xebialabs has produced a lovely periodic table of DevOps tools that covers the full spectrum of tooling, such as source code management, deployment, security testing and release orchestration. Some of the online cloud platforms are becoming extremely popular, because they offer either free or cheap entry points for teams to experiment with development patterns, including how they configure CI/CD pipelines.

Veracode is one of the security elements (in blue) represented in the table, and its job is to ensure that developers are able to scan and receive results alongside other testing tools. Providing free integration tools is therefore a key principle to promoting plug and play interoperability, giving the development team the maximum amount of choice.

Jump, but don’t boil

Our advice is to be pragmatic. Be ambitious but realistic about what can be achieved in each ‘jump’. Take people with you by explaining the changes that need to be made and how these might be achieved. Don’t boil the ocean and try to automate everything in one step.

Anticipate (and perhaps, even embrace) failure by being willing to address feedback so that the next iteration will be perfect. A really useful read on that point is Epic Failures in DevSecOps, it’s free and provides insight into failure and potential success. Lastly, collaborate and try to future proof the decisions you make. For example, automating everything with the exception of security will only mean that you end up shipping vulnerable software more quickly.

Veracode’s Farrington — knows his code pipeline… and his onions, presumably.