Commerzbank creates Hadoop-based platform for business-critical insights

Past experience helps technology choice

Tomak’s time in Silicon Valley gave him experience with the kind of technologies banks are just beginning to look into and implement.

For example he selected the Hadoop open source environment to build the platform, as a result of his past experience with the technology. “I am kind of biased based on my background in Silicon Valley. I worked with Yahoo which created the environment that became Hadoop. It was an internal system at the time but then was made open source,” he said.

“Over the years, I have built Hadoop environments for other companies in Silicon Valley.”

But bringing such a platform to a business outside the tech sector involves making changes to the way it is used. To this end Commerzbank’s central platform uses multiple technologies on top of Hadoop to help users that are not tech experts.

“At Yahoo the work was very technical,” said Tomak. “You had to write your own processes and scripts to use the data.”

“It would take days or even weeks to put together a unified view of the data or a report that a line manager of executive needed. I was looking for fast and visual tools to enable us to do things much faster than writing code,” he said.

Refined data

This is where software from Trifacta came in. The self-service data preparation platform allows the people that best understand the data to take it from raw to refined. This is done by turning it into a user experience problem and using machine learning to automate a lot of the complicated work that used to require code.

The people that understand the business context are able to get in and work with big and complex data sets and blend and structure the information.

Enabling more than just the data scientists to use the system is vital as there is a wide range of potential users. For example, they range from the highly tech savvy data engineers and data scientists to business analysts, risk compliance officers, and fraud investigators.

The Commerzbank data platform provides a self-service for all the bank’s departments. “We are working to get people from across the bank to use the central platform to gather the insights they need,” said Tomak, adding that this can be ad hoc.

Recruitment challenge

But there is a major skills challenge when it comes to getting the right staff to meet the demands on Tomak’s team. There are currently about 50 people in the team, but this will increase to 100 by the end of this year.

This is not easy because the candidate profile needed is changing. “The skills are shifting from pure coding skills to really understanding the business context,” said Tomak. “If you really want to solve a business problem you have to understand the business context as well as the data.”

“These are the skills I am hiring now. But getting them is always competitive because every industry is looking for them,” he said.

More to come

While a lot has already been achieved through the centralise platform there is an opportunity to scale it up across the bank.

“We have the environment up and running, we have collected about 150 terabytes of data, we have Trifacta in the environment to check data quality and provide use cases for auditing and fraud detection, but there is more to come,” said Tomak.

Going forward, Commerzbank will continue to collect the data from different parts of the bank, harmonise this data, keep creating more automation and also create an engine on the data lake using tools to streamline processing of the data and creation of insights.