CI/CD series - PagerDuty: the pipeline is only as good as you build it

Stratton writes…

We need to remember that CI/CD is not just Agile iteration, but a direct comparison isn’t quite right either – it’s a little bit like asking for the differences between a car and an engine.

CI/CD can definitely enhance and support Agile methodologies, but it’s not directly linked. You don’t necessarily need to be doing Agile in a structured way to take advantage and get benefits of CI/CD — and CI/CD is a great way to support feedback loops – but they aren’t required.

That’s not to say I don’t support or endorse good feedback loops – they are critical to software success. But they’re slightly orthogonal to CI/CD and shouldn’t be conflated.

You can absolutely do CI/CD without feedback loops – CI/CD is about how changes are tested and deployed. Your CI/CD implementation will still “work” even if you aren’t getting feedback – it’s about how the software is built and shipped. I can write a great CI/CD pipeline that has nothing to do with feedback, and it will work well and accomplish its goal, which is to ensure that the software is shippable at any given time.

No plug-&-play CD/CD

Nothing in CI/CD by itself will provide fewer bugs.

You can’t just ‘install some CI/CD’ and increase quality. Having useful and high quality tests as part of your pipeline are what will help reduce bugs – or at least make those bugs cheaper to fix. But bugs will happen. We want to detect them as close to the introduction of the bug as possible and CI/CD provides the ability to do so; but it won’t do it ‘automatically’. You need to include proper tests (including security tests for example) at all stages of the pipeline… and as early in the process as possible. Your pipeline is only as good as you build it – and continually reviewing it just like you would review your application code. If a bug escapes into production, part of the postmortem on the incident for that bug should include any potential improvements to your deployment process to address catching it earlier next time.

So how frequent should CI/CD cycles be? Well, how frequent is an easy point to answer – because every commit should trigger the pipeline. Smaller batch sizes always help.

That being said, executing your CI/CD pipeline doesn’t mean you release/deploy to production! But if you have rolled up a week’s worth of changes into one merge, that makes it a lot harder to see what in that change caused the issue. But there is no magic frequency number – you should be able to deploy/release as quickly and as often as your business requires. CI/CD is about ensuring that all changes are shippable at any given time, and they are available to be released to production when the business requires it.

A question of concurrency

If your deployments and builds required a lot of computing and are taking longer than you would like, looking at ways to parallelize them is definitely helpful, but by no means required. I would caution folks against over-optimizing for an issue until it actually comes up.

Start small and scale when you do hit the cases that require you to scale.

I find that I reiterate the distinction between how CI/CD contrasts with continuous deployment quite often. The only difference between continuous delivery and continuous deployment is whether the last step – the push to production – is either automatic or a human gate. Not everyone is ready for continuous deployment (or may never need it!) but everyone can, and should, engage in continuous delivery. We should be testing the deployment of every change, just like we perform functional tests of every change. That’s continuous delivery.

It’s also important to note that the difference between functional and unit tests has nothing to do with whether they exist in CI/CD or elsewhere. The difference is in what is being tested.

That being said, unit tests tend to exist in the “CI” part; which is to say they are testing the code without it being deployed somewhere. The functional tests require the application to be running, which is more of the “CD” side of the equation.

A document trail is as valuable as it’s readability by others and also how often it is actually looked at.

Stack traces aren’t always helpful for someone other than the engineer who wrote the code – having your tests provide meaningful, human-readable output is key.

PagerDuty’s Matty Stratton: knows his code, all the way to the motherlode.