The state of agile software development in the UK

Agile has been in finance for years

The finance sector has been doing agile for years but for different reasons, according to Mike Burrows, agile expert at management consultancy firm David J Anderson & Associates. Burrows previously worked in the investment banking sector.

“It could be said that the banks - the investment banks especially - have for long periods been busier competing with each other than focusing on their customers' needs. But whatever the driver, they have long understood the value of fast delivery and feedback. Flexible architectures - for example distributed systems connected by middleware - have since the 1990s enabled business-aligned teams to release functionality at a pace that suits their respective markets," he said.

"Behind closed doors, unit testing and other kinds of test automation, continuous integration, automated deployment and so on were well established [at banks] quite some time before the Agile Manifesto was written.”

Because of distribution channels such as the web, mobile, and dedicated connectivity, the slightest mistake is widely visible very quickly, said Burrows: “Now more than ever, banks need architectures and processes that can respond very quickly. In that sense at least, the need for banks of all flavours to pursue agility is unlikely to go away.”

He also said businesses adopting agile development techniques need to get used to failure.

“Agile could be said to mean accepting the truth that we can make progress with incomplete information. That makes the whole process - not just the specification part of it - dedicated to the process of knowledge discovery. Once we understand this, failure is neither shied away from nor pursued recklessly, but accepted as part of the learning process.”

Agile is not just for business

The UK public sector is as conservative an organisation as can be. Spending is closely scrutinised as is the return on investment. But the government is currently pushing the use of agile methodologies for its digital government initiatives.

Jose Casal from the Agile Methods Specialist Group at the BCS said public sector organisations face two main challenges to adopting agile thinking: “One is the heavily hierarchical command-and-control nature of the organisations, while the second one appears to be the big-bang approach to procurement that dominates the public sector.”

He said the key to a successful introduction of agile in the public sector could be to try not to push agile at all: “If we end up in a situation where agile is enforced as a silver bullet, the likelihood is that that agile will fail to deliver any sustained benefit to the organisation.

“Instead, we need to focus on delivering increased business value, arm ourselves with a lot of patience and go as slow as the organisational tolerance for change,” added Casal.

He said there are “exciting” examples of agile working in the UK public sector such as the Gov.uk website, the Government Digital Service (GDS), the work being done by DVLA and NHS Choices.

Agile skills in the UK

Agile is a different way of working and a lot needs to be done in the UK to ensure there are enough people with the right skills, according to Casal.

He said if you use LinkedIn as a measure of the agile skills base in the UK there would appear to be no shortage. But he said, “The reality is, perhaps, not so encouraging.”

“I have visited companies where the agile development teams had no opportunity to apply the most basic engineering practices. We have all heard tales of completely prescriptive adoptions of Scrum where self-organisation, inspection and adaption were nothing but a distant concept. Agile cargo cults, where teams follow the practices without being aware of why those practices are there, are easy to find," said Casal. 

He said many organisations are doing incredibly shallow agile adoptions: “Deep agility in the UK mainstream market is probably a decade away and to help bring it forward we need to go back to the fundamentals of agile and ensure that people understand why agile is what it is, rather than a passing fad. This is what BCS is addressing with the introduction of its BCS Agile certifications.”

James Lomas, CIO at Comparethemarket.com, said the company’s reliance on agile development has led it to build specific training programmes for development staff.

“We work with individuals to develop not only their technical skills, but behavioural aspects like how to give and receive feedback, run a stand-up and operate our product team's 'build it – run it' philosophy.  The idea is that in 12-18 months individuals will be contributing and fully paid up members of one of our product teams,” he said.

Lomas said that Comparethemarket.com is working also work with universities to create a syllabus that reflects the needs of an agile workplace. “This means that students are now learning techniques like test-driven development and continuous delivery, event sourcing and NoSQL databases,” he said.

“These sorts of initiatives may never give us any direct payback, but enables us and like-minded companies to bridge the gap in the future."

Where should agile be delivered from?

If you cannot find the skills nor justify an in-house resource, deciding where to outsource is not as simple as looking for the best price. All suppliers in all locations would be committing commercial suicide if they did not at least claim to do agile, but businesses must scrutinise what they will get from different suppliers in different locations.

Many large enterprises have outsourced their application development to offshore locations such as India to cut costs, but Carl Bruiners, agile consultant at home furnishings retailer Dunelm Mill, said working in an agile framework challenges the practice of offshoring.

"Often I have found in various organisations the decision to offshore to low-cost regions is often not the choice of the company's IT function, but the choice of the finance department," he said.

He recommends IT departments engage with the executive to explain the cost of ownership and the value return. 

“It is often narrow sighted not to consider the other factors you inherit when offshoring - language difficulties, communication latency, culture, etc,” he said.

Bruiners said during his time working for GE, an offshore engineer was a third of the cost of an engineer based in Cambridge, UK, but the Cambridge engineer was four times more productive and therefore far cheaper per feature delivered.

He does not see much difference between the locations that services are delivered from. “In practice I have never found there to be a big difference between onshore, nearshore or offshore. While at GE and Lloyds I had teams based in Cambridge / London and ones based in India. At Dunelm I have teams in Leicester and London, yet the communication challenges remain the same as those at GE and Lloyds,” he said. 

“If you have separated your delivery capability you need to ensure that you leverage as much as possible on the various communication / collaboration tools available - instant messaging, video conference, interactive whiteboards - to try to bridge the physical gap between teams.”

He recommends that businesses using agile methods should try not to split the team between two locations but to create discreet teams in each geographic location to deliver features. 

“For example, when I arrived at GE an agile team could consist of two developers in Cambridge and two developers in Hyderabad, India. This created numerous problems. Communication latency - even effective stand ups become a challenge - and effective pair programming was virtually impossible.” 

He created discrete agile teams in India and Cambridge so that each team would deliver a feature in its entirety: “This improved our effectiveness as team communication issues were localised and the behaviours of the teams improved as each team felt more inclusive of delivering a great product.”

The future of the agile methodology

With the agile methodology increasingly the norm there are inevitable questions regarding standardisation. But Jose Casal at the BCS said the idea of a more standardised methodology is reminiscent of The Lord of the Rings: "One method to rule them all".

“When considering the possible need to formulate a new standardised methodology, we must remember that agile is firmly based on the empirical process and, as such, any successful implementation of agile will need to be carefully attuned and responsive to the contextual characteristics of complex organisations,” said Casal.

“We could argue that the Agile Manifesto already delivers this and that we have great methods and frameworks at our disposal. XP (extreme programming), Scrum, Kanban, AgilePM, etc can offer what we need to work in agile ways, if they are used - rather than abused - in ways that are sympathetic and align to the existing business context," he said. 

"Essentially, I don't think it is possible, nor advisable to create a ‘one size fits all’ agile method that will work in every business environment.”