Databricks bids to marry AI and BI

Data warehousing gave birth to Business Intelligence.

This was a point memorably made to me by the analyst Mike Ferguson when I interviewed him for one of our articles celebrating the fiftieth anniversary of Computer Weekly.

He said, in 2016: “Data warehousing had to happen and was absolutely aimed at the BI market. Up till then all we had were those green and white printed sheets, spewed out of transactional database systems, to report from”.

There was a triad at work then, in the 1990s and 2000s, of data warehousing, ETL and business intelligence software that meant a radical step change in data analytics.

We are just two years away from the sixtieth anniversary of Computer Weekly, and a similar point could be made about what looks like a new generation of Business Intelligence. Call it what you will. Neo-BI, perhaps, or Generative AI-driven Business Intelligence. But there is a logical homology: as Data Warehousing was to Business Intelligence (think Cognos, Business Objects, Micostrategy, as well as relatively new vendors like Qlik, Tableau, and Thoughtspot) so a decade and a half of Hadoop data stores, data lakes, and even data lakehouses are to an emerging new generation of BI – and Knowledge Management. Driven by Generative AI.

Databricks has gone for “AI/BI”, as its term for a “new type of Business Intelligence” emanating from its data platform. At its recent Data+AI Summit (once known as the Spark Summit) in San Francisco it unveiled this. Part of that is a conversational interface “AI/BI Genie”.

The provider asserts its product will “democratise analytics and insights for anyone in an organization”.

On its account, Databricks AI/BI features a brace of complementary experiences: “Dashboards, an AI-powered, low-code interface for creating and distributing fast, interactive dashboards; and Genie, a conversational interface for addressing ad-hoc and follow-up questions through natural language”.

Both are said to be driven by a compound AI system that “continuously learns from usage across an organisation’s entire data stack, including ETL pipelines, lineage, and other queries”.

Compound AI

By a “compound AI system”, Databricks’ co-founder and CTO Matei Zaharia and his co-thinkers means “a system that tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools. In contrast, an AI Model is simply a statistical model, e.g., a Transformer that predicts the next token in text.”

And just as original Business Intelligence delivered data in the form of reports to business professionals, in finance, HR, operations, sales, and so on, so AI/BI is meant for a broad group of business users. Databricks is well known for being a technical provider, founded by academics and beloved of data engineers, ML Ops engineers, data scientists, and the rest – hardcore techies.

Ali Ghodsi, co-founder and CEO at Databricks said in the press statement announcing AI/BI: “A truly intelligent BI solution needs to understand the unique semantics and nuances of a business to effectively answer questions for business users.

“We believe this requires a different approach than how BI software has been designed in the past — one that places an AI system at the centre of the architecture and is designed to take advantage of the AI systems’ strengths as well as complementing their weaknesses to tackle the challenges of understanding and learning these nuances. The launch of AI/BI is a step towards building such a system”.

In the same statement, Felix Baker, head of data Services, Sega Europe added support. “At Sega, we aim to entertain the world with creative, innovative experiences, and data intelligence plays an important role in achieving that goal. We’re using Databricks AI/BI to help decision-makers ask ad-hoc questions in real time about consumer behaviour without having to depend on our data experts to construct dashboards and queries. Now our team members can get detailed insights on game sales and gameplay data by simply asking in natural language.

“AI/BI will enable us to democratise data, increase productivity, and enhance the speed of data-driven decision making throughout Sega”.

In conversation at the event, Richard Tomlinson, director, product marketing at Databricks provided me with context for the product that is worth reproducing.

He said: “we're trying to bring together business intelligence with artificial intelligence. We've always had a way for our customers to do lightweight dashboarding on Databricks as part of the SQL offering. More and more customers started using it and liking it, so we put a lot of engineering behind that.

“Then with the Gen AI revolution, we started to think ‘what if we could redesign and restart business intelligence from the ground up using LLMs?’

“Genie, which is the chat-like interface, is five or six different LLMs. So when you ask a question, the first thing you'll see is it says it's thinking, and that's all of those LLMs having a little chat to figure out the right way to answer the question. And the other great thing about it is we've explicitly built it so that when it is less than 95% sure of its answer, it will just say, ‘I don't know. Would you like to explain to me?’ It could be a business concept like churn it does not know. So you tell it what that is, it understands it and then applies that logic the next time it’s asked”.

Databricks not alone

One of the users I spoke with, on condition of anonymity, was positive about Genie. “The whole concept of AI-enabled BI is has been the Holy Grail for us for a while”. He also mentioned other providers as being on the same track. For Databricks is not alone in trying to marry artificial intelligence – whether GenAI or classical AI – with Business Intelligence. I recently spoke with Karel Callens, CEO and co-founder at Luzmo, an embedded analytics software company, based in Belgium.

His company has introduced what it calls “an embeddable AI insights component that brings data-driven decision-making to the broadest range of end users. [Embedded] into any workflow, the component lives inside the tools and applications that users rely upon, generating business insights that are tailored to the context and goals of whoever is engaging with the data”.

In the statement announcing this Callens said: “The shift from traditional business intelligence (BI) frameworks to more dynamic, Al-driven systems represents a transformative leap in how organizations operate and make decisions, but for knowledge workers drowning in a sea of tools and information, these need to be integrated as part of their natural ways of working”.

Viewpoint

The entire traditional Business Intelligence community – whether Salesforce’s Tableau or  Thoughtspot or Qlik, and the rest – is thinking along similar lines. More importantly, user organisations are looking for that “Holy Grail” of Business Intelligence enriched by classical and Generative AI.

Recent research from Enterprise Strategy Group delves into the emerging convergence of AI and BI, in this Research Report: Unleashing the Power of AI in Analytics and Business Intelligence. The report’s authors, Mike Leone and Christian Perry state: “With the rate of change in the business often eclipsing the rate at which data can be collected and analysed, organisations need help ensuring the timely delivery of accurate insight based on the current state of the business. By enabling users to access and analyze data without the need for specialist knowledge or coding expertise, AI is helping democratise analytics for the wider business.”

That fundamental drive to democratise analytics is one that Databricks’ founders have harped upon consistently. It sems to be evident in their AI/BI product, but, as the old analyst’s cliché would have it, in terms of a broader business user population engaging with it, time will tell.

For some historical context, modesty forbids recommendation of that CW fiftieth-anniversary article, CW@50: Data management – Five decades of prospecting for business value.

Brian McKenna is a senior analyst at TechTarget's Enterprise Strategy Group, who focuses on business applications. Previously, he was an editor at ComputerWeekly.

Enterprise Strategy Group is a division of TechTarget. Its analysts have business relationships with vendors.