MoneySupermarket.com orchestrates data pipeline with GKE

Machine learning data pipeline

The various steps in the machine learning data pipeline involve data quality, preprocessing for normalisation and standardisation, feature extraction to identify new classes of data, model training, and assessments of model accuracy accuracy based on using a test data set.

The flexibility of GKE allows MoneySupermarket.com to use it for several projects, including machine learning (ML) and web-facing application programming interface (APIs) – using Python and mostly XGBoost as the ML classifier in the container application code.

Containerised data pipeline for ML

MoneySupermarket.com uses ML to serve its personalised customer recommendations, and GKE forms the backbone of the ML model training and inference pipelines.

Each task in the model training pipeline – data extraction, feature engineering, model training and model evaluation – runs as containerised applications in GKE, and is orchestrated with Cloud Composer. By using GKE to orchestrate containers, the data pipeline for ML can be parallelised.

Using containerised applications for each pipeline task allows data scientists to make frequent incremental improvements through continuous integration and continuous deployment (CI/CD) practices.

“We can try out several types of algorithms and handle different sizes of data sets, and update any stage of pipeline without impacting other parts [of the data pipeline],” he says.

MoneySupermarket.com did originally consider using a virtual machine (VMs) instead of containers to handle the ML data pipeline, but this would not have scaled well: “The downside we found is that we would have needed  to run whole end to end process on one VM,” says Atwal. “To scale up for many models and multiple customers, we would have needed a very large VM.”

Using a VM would also have meant the data model had to be processed sequentially – rather than in parallel – which is what it has managed to achieve using containerisation.

As Computer Weekly has previously reported, MoneySuperMarket.com migrated from a legacy SAS analytics platform to GCP, using Google’s serverless software components, including BigQuery, Kubernetes, Dataflow and TensorFlow.

Moving to Google has enabled the company to simplify its data architecture. Based on Google’s reference architecture, MoneySuperMarket.com has been able to deploy serverless and software-as-a-service technology, which meant there was no infrastructure to manage, enabling the data science teams to concentrate on getting their work done on GCP, says Atwal.

With its new analytics platform, MoneySupermarket.com has benefited most from the speed of development and running big tasks. In Harvinder’s own words, the most notable change has been the deployment time for its machine learning pipelines.

“We went from eleven hours down to about five minutes,” he says. That meant the models could be updated every day instead of once a week, which, in turn, led to more relevant communications and offers, ultimately helping customers to save more money.