Modern development - GigaOm: The new agility-structure-control balance in DevOps

Collins writes as follows…

I’ve recently spent a lot of time researching, discussing and thinking about Value Stream Management (VSM) –  another three-letter acronym to beset the world of tech. In broad terms, VSM brings process management and optimisation to the rapid-delivery cycles of DevOps.

Strip away the jargon and it’s a concept that has appeared in response to a need, namely that many DevOps pipelines are complex, fragmented and just a little bit chaotic. Whilst not wanting to return to the halcyon (yet cumbersome) days of waterfall methodologies, a little structure, oversight and optimisation goes a long way.

While turning these ideas over (VSM borrows yet again from lean manufacturing and a plethora of tools that providers are aligning around its principles), I’ve wondered how not to throw the baby out with the bathwater. I’m reminded of two conversations. The first, a ten-year-old discussion around Continuous Integration (CI) and Continuous Delivery (CD) with a certain Dave Robertson, an old friend and software industry stalwart who was then at Perforce.

Get it right first time

I paraphrase but the gist was thus: “Agility is not the simple option. For Continuous Integration (CI) to work, you actually have to be far, far better at getting things right first time, which means producing high quality, testable code.” Fast forward and CI is still a challenge for many of the organisations we speak to, even as they look to address higher-order ideals such as DevOps and indeed, VSM.

This notion of ‘get it right in the first place’ speaks of the all important ‘culture’ element — in this case, of being disciplined and focused from the start of any software development project — that remains so much of a challenge. If we were already good at this, we wouldn’t need other responses such as end-to-end quality, shift-left security and so on. At the same time, we don’t want to lose sight of the need for doing things at speed. Perhaps there is some resounding lesson there for what should make modern software development look, well, truly modern.

This links to another, more recent conversation, this time with Claire Vo, chief product officer at progressive delivery and experimentation tools company Optimizely, which was featured on my Voices in DevOps podcast. “Delivering software at scale is about two things: delivering faster and delivering with confidence,” said Claire, succinctly summarising the circular dilemma at the heart of Agile practice.

So then, how can that circle be squared?

Squaring the circle

The answer comes down to a two-pronged response: first who to involve… and second, what to measure.

To answer the first element, Claire strongly believes in the power of involving customers at all stages of the process, not just the end. This strategy can be scary and comes with its own jargon (co-creation, anyone?) but the organisations I’ve spoken to, from big banks to hospitals, have seen strong success from this approach. So really, we could suggest that co-creation of software is the ‘modern way’ in the context of the discussion in this series.

From Claire’s perspective, this enables experimental approaches that can still deliver. “We refer to it as progressive delivery: incrementally rolling out experiences to your customers, testing them and then rolling them out further. Data driven, customer centric development needs to work like this,” she says.

The second element (what to measure) goes to the heart of what we mean by Value Stream Management.

From the traditional… to the modern

Essentially, most (traditional) project approaches favour the project — once requirements have been defined (or epics and stories, whatever your terminology), all focus moves to delivery, whether or not the things being developed will meet the need.

Meanwhile, Value Stream Management favours the outcome, the benefit, the upside, the good that will come out of a project — which is much more modern, isn’t it? This implies another cultural shift, which is to collaborate, test and learn, as Claire puts it: “Good developers should be thinking: how do I verify with customers that this works?” Of course this can also be scary — who likes asking for feedback?

But such short term pain can be empowering and can increase speed rather than slowing things down.

I remember once (back in UML days), I was teaching people how to develop software by asking for customer input: the client was a digital microscope firm. “That’s fine, said an attendee, who happened to be the project manager,” we have an ex-customer in marketing, let’s interview her.

So, with trepidation, we did. And we found that eighty percent of the product features were unnecessary, at least on the first pass of using the microscopes — as this pass would likely be by researchers looking for pictures to attach to their funding requests. Finding this out completely transformed the user interface design and feature planning.

So, yes, there is a way to stay Agile but add structure, embracing flexibility without losing control. The answer is a relentless pursuit of value, as dictated by actual customer needs, measured in real-time. Now, there’s a cultural shift to modernity for software development worth having.

About Jon Collins, VP of research, GigaOm

Collins has advised the world’s largest technology companies in product and go-to-market strategy, acted as an Agile software consultant to a variety of enterprise organisations, advised government departments on IT security and network management, led the development of a mobile healthcare app and successfully managed a rapidly expanding enterprise IT environment. He is frequently called upon to offer direct and practical advice to support IT and digital transformation strategy and has served on the editorial board for the BearingPoint Institute thought leadership program. Collins wrote the British Computer Society’s handbook for security architects and co-authored The Technology Garden, a book offering CIOs advice on the six principles of sustainable IT delivery. He has written innumerable papers and guides about getting the most out of technology and is an accomplished speaker, facilitator and presenter.

Image Source: GigaOm