Whole-system view of railways from Resonate on Redis offers progress

The railway industry is on the move – from moving metal to taking a whole-system view that also takes passengers into account. Although not as well-known as its principal customer, Network Rail, Resonate, a software supplier to the railways, is playing a role in that transition, says Daren Wood, its vision director.

And Jon Gregory, application architect, and his technical team are delivering an important piece of that. They have built an event-driven microservice application with Redis Enterprise at its core, from key-value, in-memory open source database supplier Redis Labs – now called Redis. The application uses Redis Streams and Java Microservices to provide real-time processing of train movements to a browser-based client using SSE connections to push the updates.

Resonate provides traffic management technology for the railways. Wood says its technology is “all about how to operate the railways more efficiently and deliver a better customer service”.

He adds: “We deliver systems that take data on planned activity and real-time activity on the network, present it to Network Rail’s operational teams, so they can make better decisions, improve the quality of their plans, [in the context of] a lot of changes.”

The railway industry is very fragmented, says Wood, with the train operators on the one hand and Network Rail on the other.

“We’re doing work over this year to join up systems across the industry,” he says. “For the first time, some of these people will have shared views of all those resources – tracks, rolling stock, crews. The project began in March 2020 and is ongoing.”

Gregory, as the lead application architect, says: “We receive data from a lot of different systems which weren’t always designed to work together, so we need to combine it before we can then present it to the users [at Network Rail].

“That drove the need for real-time, in-memory processing. But alongside that we also needed the system to be hosted anywhere, on premise and [with AWS] in the cloud. We needed high availability and replication, which is where Redis stands out against a lot of other in-memory databases.”

The project required not only real-time processing, he says, but also the ability to “go back and replay the data, replay multiple streams of that data so you can compare different scenarios on the railways”.

“And so we are using Redis Streams because that creates an immutable record of events or things that happened. As we were going down that path, we realised that with Redis Enterprise, we have one tool that solves a lot of problems: streaming, database performance, replication, and a database that we could install anywhere. We have now a system where we’ve got not only the real- time view of what’s actually happened on the rail infrastructure, but also what has happened in the last 30 days.”

The technical team also looked at [messaging system] Kafka, but decided it was “not the right fit for this problem, mainly because of the complexity”.

The ability to keep the system compatible with running on-premise was also important, says Gregory. “We have a lot of data on the railways. And if you want to keep 30 days available for replay, you would soon get into a difficult situation [with public cloud providers], because you can only keep, say, 10 or 14 days within a cloud provider because they apply restrictions.

“We were trying to avoid a situation where we put a ceiling or constraint on the system. If you were to build a system that was dependent on Azure or AWS, providing 30 days on-stream, but then you’re too small to influence their decision if they decide to drop down to 15 days, it would essentially break your system.”

The system being replaced is about 25 years old. Wood says: “The first phase of this project is more or less like-for-like replacement in terms of functionality for the visibility that they’ve currently got across their network. But with the challenges the rail industry’s got emerging from the pandemic, there is that need to be more flexible, and to get more out of the [rail] network, responding to demand.

“So, we’ve put in place a whole set of infrastructure in the broadest sense that can enable all sorts of reuse of these streams of data. We’re just starting with this relatively simple visualisation of what’s going on the network, but we can now build in other data streams, and build up a much more complete picture of what the real-time railway is doing. Potentially, we can start building in all sorts of predictive analytics and other tooling.”

Wood adds: “I think in the short term, these products are all about improving the efficiency of the operation, improving customer service, particularly in terms of train performance. But then there is the next evolution in building a set of tools fit for a 21st century railway, to be slightly corny about it.

“There is a big role for technology to support that refocusing away from managing the physical set of assets to managing a customer service. And Redis and AWS are the key strategic technologies that support this whole platform.”