Brian Jackson - stock.adobe.com
Weather Source uses Snowflake to keep ahead of extreme weather
The impact of Storm Ciara shows the importance of accurate weather data. We look at the tech behind Weather Source’s service
Due to the increased frequency of extreme weather, businesses have to make sure they protect themselves. In January, Børge Brende, president of the World Economic Forum, warned that climate change and environmental risks are the biggest threats facing society.
Even businesses that are not directly affected by extreme weather events may face risks if accurate weather forecasting is not built into their business assumptions, with Peter Giger, group chief risk officer of Zurich Insurance Group, recently asking: “If weather patterns change, how do I look at the probability of events occurring?”
The floods in 2013 were reported as the worst December and January rainfall on record. Two years later, in 2015, Cumbria and north-east England faced Storm Desmond, which broke records for the most rainfall in a single day. This week, residents of Cumbria are again facing a big clean-up in the wake of Storm Ciara.
Weather sensitivity
Weather Source provides enterprise clients with meteorological data to support their businesses. Such information is becoming business critical, due to the effects of climate change.
“What we are seeing is climate change is in full effect,” says Mark Gibbas, CEO of Weather Source. “Weather information is important to many companies that have sensitivity to weather, which can impact their operations and their revenue.”
As weather becomes more extreme, accurate forecasting becomes increasingly important in helping companies to quantify and manage climate risks.
Weather Source has created a business to provide such information to businesses. According to Gibbas, what makes the service unique is that it offers a continuum of weather over space and time, which provides a history of normal weather conditions for any hour of the year and the variants in weather.
In the early years of the company, around 2015, Weather Source used MySQL on optimised hardware for speed. As it expanded, it began hosting the service on Amazon Web Services (AWS) and Google Cloud Platform (GCP). While public cloud hosting is still being used, Gibbas says the company wanted to achieve greater speed and efficiency.
Due to the complex dataset – it requires a time series database which spans the planet – Gibbas says the company needed something that could handle petabytes of data.
Weather Source selected Snowflake’s Data Exchange to provide its clients with a better understanding of climate change risk and to increase its customer portfolio. This data includes hyper-local weather information and climate data from around the world.
“Snowflake is extremely efficient and fast, and has powerful technology for data sharing,” he says. “This allows us to store information once and share with many different clients, cutting down on the total storage we require and avoiding duplicating information.”
The Weather Source service runs on Snowflake, which is hosted in AWS. Moving historical data to the cloud is time-consuming, but once there, updates to the database can be achieved relatively quickly. For instance, migrating half a petabyte of historical data to the public cloud can take several days.
But once the data is in Snowflake, operationally, it only takes seconds to update the latest weather information, according to Gibbas. The Weather Source has two ways to upload data: either via an AWS S3 bucket, which can automatically load data into Snowflake, or via the Snowflake application programming interface (API), which Gibbas says is “very fast”.
“Snowflake is also very efficient computationally. Only a few hundred gigabytes of new data per day is processed,” he adds. For comparison, the company needed to process a terabyte of data on GCP or AWS.
One point of shared forecasting data
Such efficiency gains are essential in accurate forecasting. Each day’s forecast is made up of several forecasts and is projected over 15 days, leading to petabyte-scale data processing. But once the data is in Snowflake, Weather Source does not have to move the data.
The Weather Source data service is available in Snowflake’s Data Exchange market. Snowflake users who require weather data are able to access different views of the weather data provided by Weather Source, without the need to download anything.
Traditionally, organisations had to make a copy of the data locally on their own systems when they needed to share data, which Gibbas says “dramatically expands the amount of data” they need.
However, data sharing through the Snowflake Data Exchange market means the data is stored in a single place. “They just access one common database,” adds Gibbas.
This also cuts down the speed of data transfer as “there is no need to load the data into their own systems”, he says. Since there is only a single copy of the weather data, as and when the forecast is refreshed, users can immediately access the new data.
Read more about Snowflake
- Snowflake is looking to grow its data warehouse platform with new funding and a partnership with Salesforce that could differentiate its platform from rivals.
- The rise of non-relational databases has been a feature of the data management landscape in the wake of big data – but to complement or replace relational?
