CW Innovation Awards: SensorFlow reaps returns with re-architecture

Re-architecting from within

Facing these challenges, SensorFlow decided to re-architect its entire platform instead of making incremental changes. The project was split equally between internal resources and external developers. The former mostly concentrated on front-end and application programming interface (API) layers for their dashboard development while the latter handled data migration and the design of data schema architecture.

The project had three main objectives: reduce IT infrastructure cost to less than $10,000 per month, improve data quality access and decision-making agility across the company, and implement a new real-time business intelligence (BI) dashboard for its customers.

To do this effectively, the team moved the entire back-end stack from a serverless DynamoDB-based architecture to a data-centric SQL-based architecture powered by Aiven. It also redesigned the front-end user interface for both customers as well as internal users, and integrated a BI platform, Looker BI, to serve as an insight delivery platform for the company.

The entire project took six to eight months and comprised 13 people from engineering to product teams, with a project budget of between $400,000 and $550,000. The return-on-investment savings of $40,000 per month gained from this re-architecting effort was expected to be about 13 months.

Overall, SensorFlow reduced the cost required to maintain and operate IT infrastructure by 80%, the ability to implement new dashboard features within days instead of weeks, and the speeding up of decision-making based on correct data. The project also allowed departments to engage SensorFlow’s self-service business intelligence portal instead of having to rely on help from the data science team.

Other benefits gained include the optimisation of IT infrastructure headcount so that only two employees can manage the workload and the improvement of product delivery and installation due to better actionable information.

“The project saved us thousands of dollars in infrastructure expenses and at the same time dramatically improved access to business critical data across all teams,” said Max Pagel, co-founder and chief technology officer of SensorFlow. “This democratisation of data has enabled all teams to work much more efficiently and sped up decision-making significantly.”

Challenges and lessons

One of the biggest challenges of this project was to convince the company of the necessity of a full re-architecture. Initially, the real cost of the old architecture was hidden due to the availability of AWS credits. To ensure the project took off, SensorFlow built a functional prototype to demonstrate business value.

The team achieved this by building out parts of the stack on a separate analytics database in order to demonstrate the possible improved speed and decision quality, when given access to a well-structured database. SensorFlow was also able to demonstrate the value delivered from this prototype to convince management and the rest of the company of the value of such an architecture.

Another important lesson learnt was to carefully assess good vendors who can manage the change management process and to work hand-in-hand with them on the change management journey.

Lastly, to ensure long-term success, SensorFlow created a solid information model. Being thorough at the beginning of the project was also key, as any mistakes in building the model could lead to significant problems.

“Embarking on a full re-architecture is a daunting task and a decision not to be taken lightly, but when done right, it can be a massive enabler for the success of our business in the future,” said Pagel.