Kmart taps event streaming for real-time insights

Australian retailer Kmart has developed an event-driven data platform to support its move from bricks-and-mortar to online retail.

Speaking at the Melbourne leg of Confluent’s Data in Motion tour, Kmart enterprise technology principal architect Duane Gomes said that the requirements included real-time operation, an event-driven architecture, modern integration methods, and support for event streaming for applications.

Data pipelines acquire and process data from various applications and feed the data lake and analytics and artificial intelligence (AI) platforms for business intelligence and reporting requirements.

Sachin Walia, Kmart’s enterprise technology solution architect, said that the company’s engineering platforms team provides cloud and datacentre hosting, developer technology, and integration services to other teams within the enterprise technology group.

Part of the strategy is to decouple data producers and consumers, with application programming interface (API) management, file transfer and event streaming as integration methods.

Event streaming is done with Confluent Cloud, but several other software-as-a-service (SaaS) products are part of the overall architecture, including those from Palo Alto Networks, SumoLogic, GitHub, Alation, and Microsoft Azure. The Kmart Group Cloud “stitches it all together”, Walia said.

Kmart’s Confluent journey began in 2018, with the software being hosted by Kmart. The next two years saw the development of Kmart’s Sophia data platform and Chanakya data science platform, which were migrated to Confluent Cloud in 2021.

The retailer’s current Hamilton event streaming platform, powered by Confluent Cloud, was implemented in 2022, and a year later applications using Sophia and Chanakya were migrated to Hamilton.

This year’s developments include the adoption of Flink that supports stateful computations over data streams, Confluent’s stream sharing capability, as well as lineage in the form of a visual representation of where data comes from, how is it transformed, and where it goes.

Looking ahead to 2025, data will become the fulcrum at Kmart, and the company will use Flink “to work with all that data”, said Gomes. In addition, stream sharing will be used to support a more managed approach to sharing data with partners including customer loyalty programme Flybuys.

Hamilton handles more than 200 million events daily and has been well-received by its users.

“Hamilton allows us in supply chain to simplify our complex integrations with our partners and suppliers across the globe, by decoupling and moving to real-time integrations based on events,” said Rahul Varma, supply chain engineering manager at Kmart.

Jimmy Lamba, Kmart’s data platforms and automation engineering manager, added: “Hamilton has enabled us to uplift the Sofia data platform to a new posture of security and governance, allowing us to offer even more robust capabilities around data services to our customers,”

Goldi Ahuja, Kmart’s stores senior engineer, said that Hamilton has helped to scale and simplify the retailer’s transition to an event-driven architecture to support real-time initiatives for more than 350 stores across Australia and New Zealand within a short period.

For example, the platform delivers real-time inventory information to support marketing campaigns, real-time container shipment tracking, real-time stock location to facilitate replenishment, real-time sales figures, and real-time customer interactions.

This made it easier to sell the idea of the platform to non-technical management. For instance, the CEO would look at the sales figures every morning, and now they are available in real-time.

Customers also benefit from Hamilton, as it lets them see whether a particular item is in stock at their local store. If they visit without checking and it turns out that what they want is out of stock, staff on the floor can use their handheld devices to find the closest store that has the desired item.

According to Confluent’s 2024 Data streaming report published this month, 86% of IT leaders in Australia with experience in data streaming saw a two to 10 times return on their data streaming investments, and 94% of them regarded data streaming as critical or important to achieving data and information-related goals.

Simon Laskaj, Confluent’s regional director for Australia and New Zealand, said: “It’s encouraging to see that many Australian businesses are already reaping the organisational and ROI [return on investment] benefits of a complete data streaming platform.

“The importance of reaching the ‘central nervous system’ level of data streaming maturity will become more important as transformative technologies, such as AI and machine learning, become ingrained in operations, requiring a robust data and governance strategy.”