Refinitiv uses Google Cloud to revolutionise delivery of huge datasets
Financial data giant is expanding delivery of its services in the cloud, removing the need for customers to use hard disks and trucks to transport large datasets onsite
Financial data giant Refinitiv has removed the need for customers to use trucks and thousands of hard disks to receive large datasets, by using a platform build on Google’s cloud.
The organisation, formerly the Thomson Reuters financial and risk business, has moved its Tick History database into Google Cloud, which will enable customers to analyse the data there, rather than having it shipped to them on hard disks.
The Tick History database contains a large amount of historical data, with about 5PB of data on trades and quotes from more than 500 trading venues going back to 1996. It adds 4TB-5TB of data every day.
Refinitiv’s customer base includes the world’s biggest investment companies, such as banks and hedge funds, which use the data to build and test their investment strategies.
The datasets that Refinitiv’s customers want from the database are so large that many requests cannot be sent over fibre networks without huge delay. This means companies often transport data by truck on hard disks and uploaded to their servers.
Catalina Vazquez, proposition director for the Tick History database at Refinitiv, said customers are using the data from the database to design, build and test new trading strategies, applying deep analysis to them.
But the datasets are huge, creating financial and logistical challenges, she said. “Traditionally for customers, there is the challenge of extracting the data, transporting it and then loading it onto servers in their datacentre. But we are providing the data ready for query without having to move the data around.”
The data is being moved into Refinitiv’s own managed BigQuery instance in the Google Cloud, said Vazquez. “We are enabling our customers to simple query that dataset, running all the analyses and use cases they need. They can also leverage the artificial intelligence (AI) capabilities that Google has to offer.”
In the past, it took a significant amount of time for customers to gain access to this data, which is now available instantly. “It depended on the type of dataset, but some would actually have to be physically transported onsite,” said Vazquez. “If you wanted four or five petabytes of data delivered to you, you would probably have needed a truckload of tapes.”
Read more about Refinitiv
- Refinitiv CIO John Finch says finding people with the right level of data science skills is his biggest challenge.
- The London Stock Exchange has agreed to pay £22bn to acquire Refinitiv, creating a giant of global financial data and transactions.
- Alex Cesar steps up to be chief technology officer at financial service data provider Refinitiv, after previous CTO Phil Wellard took over as global CIO in September.
This method is still available if people want it, but Vazquez said it is not economically sensible and customers “might actually want to save themselves the headache of managing the huge datasets”.
“Due to the size of the datasets, we would either ship them on servers or hard disks, but the goal is to stop having to do that because of the latency this adds,” she said. “Some of this data can be sent over fibre networks, but even using financial services networks, it can take weeks to move large datasets.”
Cloud computing and other digital technology advancements are transforming how data is analysed in the capital markets sector, said Vazquez. “As the cloud delivers on its promise to make AI-based analytics more readily available, the potential of data to deliver answers that drive business performance gets ever greater.”
Vazquez said the world has changed and there is now no need to analyse data onsite, when it can actually be done remotely. “The data stays the same whichever you choose, so why would you take on all that extra cost when you can let someone else do that for you?” she said.
Security was a prime focus when building the platform, said Vazquez. “A huge amount of the time we spent putting this project together was making sure the environment we built was up to our stringent security standards.”
She pointed out that although it is a joint project with Google, the design and build is ultimately down to the Refinitiv team.
Moving the Tick History database to the Google Cloud is part of Refinitiv’s efforts to deliver more services via the cloud. In an interview last June, the then Refinitiv CIO, John Finch, told Computer Weekly that, operationally, the company was already a heavy cloud user, but it wanted to increase the proportion of its services delivered through the cloud.
Finch said at the time that about 15-20% of its products were delivered through the cloud. “While this is a large amount compared with many businesses, the proportion can be increased greatly,” he added.
“Our customers are thinking more and more about consuming capability like ours through the cloud, rather than through our infrastructure and their own,” said Finch. To ensure it can cater for customer demand, Refinitiv has relationships with Amazon Web Services, Azure and Google Cloud.
Three core businesses
Refinitiv has three core businesses. It is best known for its data platforms that go on trading floors to provide investment firms with the latest information to help them make investment decisions. It also processes large volumes of foreign exchange transactions, while its risk business supports compliance activities for big banks including anti-money-laundering and know-your-customer.
All three segments of the business involve collecting, aggregating, analysing, curating, storing and distributing huge volumes of data from a large number of sources. It collects more than 1PB of data every day and processes more than a million unstructured documents. Its data platform provides 40 billion market updates each day.
Some 400,000 people across the world use Refinitiv’s terminals on behalf of 40,000 financial firms, with about one million algorithms consuming the data without human involvement. As CIO, Finch is in charge of all enterprise technology, including product, engineering and operational IT.
Globally, Refinitiv’s internal IT team is about 5,500-strong, and about 4,000 others work via suppliers or contracting. The organisation has major tech hubs in Bangkok, Beijing, Bangalore, New York City and St Louis in the US, and has a UK tech team numbering about 600.
In August last year, the London Stock Exchange agreed to pay £22bn to acquire Refinitiv, potentially creating a giant of global financial data and transactions. The deal has not yet gone through, but is expected to close in the second half of this year.
