DevOps done right: How to break down the IT department silos without alienating developers

Own your code

On the software development side, one of the biggest challenges is getting people used to the idea of having to support the code they write once it enters production, rather than hand off responsibility for that to operations.

This is a process the team at price comparison site Compare The Market had to go through three-and-a-half years ago, when its DevOps push began in earnest, recalls the firm’s service delivery manager, Rohit Mathews.

Getting its development team on board with the idea of performing duties more commonly associated with operations proved particularly challenging, he tells Computer Weekly.

“During that journey, it was quite difficult to get our development teams to actually support and go on-call and work out of office hours, so it was a culture shift for us. We had to do it in stages, bit-by-bit,” says Mathews.

“That way of working is very different and the development teams are not used to it, and it was not something they were wanted to go for initially, because they’re more interested in writing new code.”

On-call work also had a reputation for being onerous and time-consuming, which is something the organisation knew it had to address if it had any hope of getting its developers on-board with the idea of joining in.

With the help of PagerDuty’s infrastructure monitoring tools, the organisation embarked on an overhaul of its incident alerting procedures to speed up the time it takes to resolve any issues that arise, and ensure its developers and operations teams were only disturbed if absolutely necessary.

As part of this process, the organisation took some time out to analyse the alerts it commonly receives, before taking steps to classify each one to determine how swiftly they need to be dealt with.

“We wanted the central product teams to understand, ‘this is my product, these are the list of alerts I’m getting every day, and these are the high-urgency ones that require a fast response’,” he says.

“But then there might be other alerts that happen at midnight everyday but no-one does anything about them because they’re not relevant, so what is the point in having them go off? Why not remove them, because if people are receiving an alert, it should just be because it’s actually an incident.”

Helping developers get over their fear of on-call work, and potential impact it may have on their work-life balance, is difficult, says Wilson.

“If you’re on-call and your infrastructure has problems, you’re going to be disturbed a lot,” he says.

But that does not necessarily have to be the case. If the infrastructure keeps falling over, that suggests there maybe technical debt issues that need addressing, which – if tackled – may help minimise the amount of on-call work developers would be expected to do.

Technical debt

If technical debt is less of a problem, taking steps – as Compare The Market has – to ensure whoever is on-call is not being overloaded by superfluous alerts often helps take the the sting out of things, and can help make the prospect more palatable for first-timers.

“You have to make sure you clean up that event stream, so people are only being called for something that is a real problem, because a lot of the fatigue people have [from on-call work] is because they’re woken up about something that is not really an issue,” says Wilson.

“Our philosophy [at PagerDuty] is focused on how can we use as much automation and machine learning as possible to help make it easier for the people who sit at the centre of this, so they can be more effective and bring greater benefit to the organisations they work for,” he says.

“Because it means your online environment is more available and more robust, and that has good economic consequences for businesses.”

Who’s leading who?

How receptive developers are to this change in their day-to-day responsibilities sometimes comes down to who is responsible for pushing the DevOps agenda within their organisation, says Ben Connolly, head of digital IT at mobile operator Vodafone.

If it is a grassroots movement, driven by the developers and operations staff that can make things easier, he says, because they will have already – one presumes – bought into the concept.

“It usually starts with the engineers wanting to be more agile, or operations guys wanting to be part of the DevOps movement. And that’s great, and it often takes off that way, and you do improve, and you do get faster, or more iterative,” he tells Computer Weekly.

Difficulties may arise if the push to embrace DevOps is coming from the top-down, which is why it is so important for organisations to think about how the move is being communicated to the people who will be on the forefront of delivering on it, says Wilson.

Developers do not want to hear their organisation is simply doing DevOps because it is the “cool or trendy thing to do,” he says, and will want to know how the move will benefit them.

That means emphasising the benefits DevOps stands to bring, in terms of delivering a better experience to the organisation’s customers or making better and efficient use of the resources the organisation has.

DevOps seeing results

With its focus on speeding up the time it takes for new features to enter production, developers in DevOps also get to see the work they do start to benefit users far more quickly, which can be hugely motivating, says Connolly.

“We’re reducing the proximity between the developer and the customer, whereas traditionally they’re weeks apart, and with [lots of different teams] between them, and in doing that, we’re suddenly giving people much more exposure to the outcomes they’re producing,” he says.

“And they take more pride in what they do because they can see the outcomes, which again makes them want to do even better.”

The idea is not to turn developers into operations staff. Instead, DevOps seeks to encourage the two groups to work together in a more collaborative way, so the resulting code can enter production with the operation team’s requirements around stability already factored into its design. The idea being the code will be produced more quickly, and will end up being less buggy and more secure.

At Vodafone, Connolly’s team has gone from pushing out code changes every six weeks to everyday since its DevOps push began, which in turn gives its software engineers more opportunities to innovate and test new features.

“This means they can try something out and see if it works, and one-in-five might stick and the others will go into the learning pile – but some of the really cool stuff we’re doing has started life that way,” he says.

Exercising creativity

Giving developers opportunities to exercise their creativity can prove an enticing prospect to resistant-to-change individuals, and – in time – soon becomes second-nature to them.

“DevOps is, fundamentally, about pace and ownership, so if we enable those things and let them do it naturally, then it does start to become self-propelling within the organisation.”

And, in organisations where securing buy-in for DevOps at the grassroots level is still proving problematic, seeing how other groups in the company have taken to the changes can be powerful motivational tool, adds PagerDuty’s Wilson.

“What happens then is other people see [what’s happening], they find it interesting and can see how it is working, and start thinking about what they can do to make it work for them – so take-up becomes viral. That’s how the DevOps movement has evolved,” he says.