Building distributed systems requires effective developer teams

When building a web app in previous years, it was common to have a server in a centralised datacentre that would be able to run your application. As usage grew, you would address scalability bottlenecks as they came up.

Nowadays, web apps are being built to scale from the start. Code is increasingly run on serverless platforms, in isolated virtual sandboxes that may well only exist for however long it takes to send a response back to the user. File storage and databases are increasingly managed for developers, without them ever needing to configure their own hardware.

One of the advantages of this shift is that code can live at the network edge, at internet exchange points that connect consumer ISPs to cloud hosting providers, allowing for low-latency loading times. This change necessarily means that code lives in multiple servers around the world from the point it is first deployed, rather than when scalability is needed. 

Software systems that, years ago, we would have built to be centralised, are now distributed systems. These practices are even making their way into core datacentres, with technology such as Kubernetes being deployed to scale up applications automatically in virtual containers.

Over recent months, it has become apparent that some technology organisations are struggling to cross this divide, in particular, companies where software is managed as a central monolith (often purported to be a “monorepo”) without well-defined communication structures between the different parts of the system. Distributed systems are effective in simple systems with scalable communication structures.

Conway’s Law is a well-known adage in software engineering management that states: “Any organisation that designs a system (defined broadly) will produce a design whose structure is a copy of the organisation’s communication structure.”

In other words, your organisation design ultimately reflects the architecture of your software. Low-trust, centrally managed organisations will struggle to build distributed systems with effective communication structures. 

To succeed in building distributed systems, you need skilled engineers who are managed effectively and motivated with the right incentive structures. This means that a culture of experimentation and psychological safety lies at the heart of building web services on the next generation of cloud technologies.

Junade Ali is an experienced technologist with an interest in software engineering management, computer security research and distributed systems.