Laurent - stock.adobe.com
Building enterprise-grade AI: Sberbank and AI Telekom
Machine learning and artificial intelligence are growing beyond adolescence to prove business value. Practical applications at Sberbank and AI Telekom suggest getting machine learning data in one place, where data scientists can experiment
It might hold about 30 petabytes of data, but Sberbank is a financial institution that most people in the West know little about. The leading Russian bank has about 100 million individual customers and three million corporate clients, and boasts net profit of around $10bn.
Sberbank built a machine learning and artificial intelligence (AI) pipeline and architecture to help industrialise earlier efforts at machine learning and data science – resulting in 50% of its models getting into production in one day, rather than the seven months it took previously.
Meanwhile, Austrian telecoms company A1 Telekom has developed cross-functional teams to build use cases for machine learning. Both companies presented at Teradata Universe, the data warehouse supplier’s annual conference, held in Denver, US in October 2019.
The companies’ experience provides lessons, as many organisations struggle to bed AI and machine learning into their business. A recent study from 451 Research found that only 20% of organisations have, to date, implemented some form of machine learning software in one or more parts of the business. A further 20% are at the proof-of-concept stage.
Sberbank’s main goals were to improve customer experience and to create new growth for the business using machine learning, says Oleg Sedelev, executive director, machine learning implementation, in its corporate banking arm.
The bank uses a complex data infrastructure, based on Hadoop data lakes, and Teradata and Oracle enterprise data warehouses. It also runs some legacy database systems.
The first question was to find out whether it could build a production-ready data science pipeline on existing heterogeneous systems. The alternative was to build a single data science system. It opted to use the existing system, says Sedelev.
One example of how Sberbank has built its machine learning pipeline comes from the business problem of “next-best action” in cross-selling and upselling to corporate clients. The bank already applies machine learning in this area and wanted to improve its performance.
The data for building machine learning models comes from two sets of sources: first, structured account and transaction data from Oracle and Teradata data warehouses; and second, unstructured data from voice recordings of client calls, plus external data social media feeds and demographic data.
Read more about enterprise AI
- OpenStack Foundation will tackle infrastructure barriers to enterprise AI adoption.
- Defining enterprise AI: From ETL to modern AI infrastructure.
- Enterprise AI adoption on fire, but foundation lacking for sustainable AI.
Extract-transform-load functions for unstructured data are performed in Hadoop, while both structured and unstructured data are transferred to a Hadoop-based data lake for building offline machine learning models. Sberbank uses a 500-CPU cluster to transcribe about 30% of .wav voice recordings of corporate client calls into text, which is converted into meaningful sentiment by a “word-embedding” machine learning technique popular in natural language processing.
After machine learning models are tested and verified, a middle layer of application processing completes customer scoring, designs campaigns and integrates output into front-end call centre, website and customer relationship management (CRM) systems.
So far, sticking with a heterogeneous approach to the technology stack has proved valid for the bank, says Sedelev. “The answer is to work with what we have – it might be several years before we update it,” he says. “Hadoop is a cheap and safe place to store lots of data. Teradata is better for some populations. It is reasonable to work with both of them and this is not something we are about to change.”
The resulting applications produced an 8% increase in targeted sales to corporate clients, he says. “I see a high potential of unstructured data for traditional corporate problems. For example, the case of using transcription of voice calls and NLP [neuro linguistic programming] data can really improve models for next-best action problems. We have seen it in practice now.”
Data marts in Hadoop
Rather than changing its technology stack, Sberbank boosted machine learning productivity by gathering all its data into one place – data marts in Hadoop – and providing their data scientists with automation tools, including H2O.ai.
It aligned its efforts in data engineering by gathering eight years of data attributes – features of data that need defining, editing and aligning before building machine learning models. The bank’s machine learning team then made 2,200 data attributes available in a searchable database to avoid duplicated effort.
“We want to be more human-centric for data scientists, so we aggregate data and make the most popular tools freely available,” says Sedelev.
The effort reduced time to market for 50% of the data models from seven months to one day, he says. “That’s a really dramatic change, and we attribute it to having all the data marts in one place. The message is clear: no data, no machine learning.”
As well as next-best offer, other use cases address customer experience, client manager efficiency, client plans and pricing.
Efforts from A1 Telekom and Sberbank to create a more enterprise-ready approach to machine learning come as suppliers and analysis debate changing enterprise data architecture to meet these needs.
Claudia Imhoff, president and founder of research company Intelligent Solutions, says: “In the last five years, we’ve all had to rethink our architectures because there is no longer a lovely tidy world where all analytics are in a single data warehouse. Those days are gone. The data warehouse is still there, but there are other areas where analytics are taking place.”
Imhoff advocates the creation of an investigative computing area, with light data governance that allows data scientists to create machine learning models from more or less raw data using tools of their choosing. But machine learning models should not go into production from this platform, she says – they should be moved out into a production environment with tighter governance. But both sides should be able to work together, she adds.
Data scientists and IT professionals in collaboration
Teradata is advocating a similar architecture, with a light-touch experimental area for developing machine learning models. Its chief technology officer, Stephen Brobst, says it is vital that both data science and IT professionals work as a team when getting models into production.
“If the data scientist creates a great idea using some dodgy open source software that does not scale and has no support, it is probably not going to go into production, and IT will figure out another capability,” he says.
“But it can’t be a case of a data scientist throwing things over the fence and some IT professional catches it and says: ‘What is this Python code? I don’t understand anything!’ And then the IT person reverse-engineers it and rebuilds the whole thing, which takes nine months. Nobody is going to wait for that.”
Teams should take inspiration from the DevOps approach to software development, he says. “From the beginning, you should create these cross-functional analytics-ops teams, taking the concept of agile and DevOps and putting them together in a data-centric context, rather than a software-centric context.”
Machine learning and AI are growing out of their teenage years and need to prove their value to the business at an affordable cost. Lessons from users, analysts and suppliers suggest you need to get all the machine learning data in one place, where data scientists can experiment with it.
Businesses should also eliminate repeated effort where possible and give data scientists free access to the tools they need. They should also consider a DevOps-style approach to get machine learning models into production.
With machine learning providing considerable business benefits, the race is on to make it enterprise-ready.
Case study: Developing business-ready use cases key to machine learning efficiency for A1 Telekom
Building cross-functional teams to develop use cases for machine learning is a vital element to gaining business benefits from machine learning, according to Reinhard Burgmann, head of advanced analytics and AI at A1 Telekom.
The telecoms firm, which has operations across eastern and central Europe, uses machine learning in financial analytics, credit ratings and next-best-offer sales promotions.
Burgmann says the group has found success in developing machine learning models that rely on teams working across functions. Although it has separate advanced analytics, business department and external partner teams, members of these teams come together to form “focus groups” to develop machine learning use cases.
“In machine learning, the algorithms are not the problem,” he says. “The data scientists know which classifications work for which problem. What is more important is finding use cases where the past is a good indicator of the future. And is the data you need available? What is the distribution quality, the amount, and do we have the history? These are some of the lessons we’ve learned.
“The enabler for the organisation in applying machine learning was catering for business-to-business and consumer enterprises, and also working with external partners, building what we call focus groups – cross-functional teams that work on developing the use cases.”
The teams include data scientists who sit within business units, as well as machine learning and data engineers from the advanced analytics team.
To understand how machine learning will be applied to business in practice, A1 Telekom uses process mining tools from Celonis. These tools gather data from business systems to describe how processes are performed in real life, as opposed to how they have been designed.
Use cases for A1 Telecom applying machine learning include demand forecasting mobile phone sales through its retail network, preventing churn from network subscribers, and understanding the net promotor score – the indicator that describes whether a customer would recommend its products or services to a peer group.
The central analytics group also works with external partners to monetise its data assets. Retailers and travel groups buy anonymised location data to understand how mobility might affect their businesses.
In its technology stack, the A1 Telekom analytics team has 80 terabytes of on-premise Teradata data warehousing and a 45-node Teradata data lake. It also uses Cloudera and Kafka for ingesting real-time data, as well as Hadoop for batch data. It is experimenting with Amazon’s S3 storage on-premise, but has no plans to move away from Hadoop, says Burgmann.
