Data Alone Is Not Enough: The Evolution of Data Architectures

Data, data, data — it’s long been a buzzword in the industry, whether big data, streaming data, data analytics, data science, even AI & machine learning — but data alone is not enough: it takes an entire system of tools and technology to extract value from data.

A multibillion dollar industry has emerged around data tools and technologies. And with so much excitement and innovation in the space: how exactly do all these tools fit together?

This podcast — a hallway style conversation between Ali Ghodsi, CEO and founder of Databricks, and a16z general partner Martin Casado — explores the evolution of data architectures, including some quick history, where they’re going, and a surprising use case for streaming data, as well as Ali’s take on how he’d architect the picks and shovels that handle data end-to-end today.

Ali: It kind of started in the ’80s. Business leaders were flying blind, not knowing how the business was doing, waiting for finance to close the books. This data warehousing paradigm came about where they said, “Look, we have all this data in these operational data systems. Why don’t we just get all that data, take it out of all these systems, transform it into a central place, let’s call it a data warehouse, and then we can get business intelligence on that data?”

And it was just a major transformation because now you could have dashboards. You could know how your product was selling by region, by SKU, by geography. That has created at least $20 billion market that has been around for quite a few decades now.

But about 10 years ago, this technology started seeing some challenges. One, more and more data types, like video and audio, started coming about, and there’s no way you can store any of that in data warehouses.

Second, they were on-prem big boxes that you had to buy. And they coupled storage and compute, so it became really expensive to scale them up and down.

And third, people wanted to do more and more machine learning and AI on these data sets. They saw that we can ask future-looking questions. “Which of my customers are going to churn? Which of my products are going to sell? Which campaigns should I be offering to who?”

The data lake came about 10 years ago. And the idea was, “Here’s really cheap storage, dump all your data here, and you can get all those insights. And it turns out, just dumping all your data in a central location, it’s hard to make sense out of that data that’s sitting there. As a result, what people are doing now is they’re taking subsets of that data, moving them into classic data warehouses in the cloud.

So, we’ve ended up with an architectural mess that’s inferior to what we had in the ’80s, where we have data in two places, in the data lake and in the data warehouse, where the staleness and the recency is not great.

In the last two to three years, there’s some really interesting technological breakthroughs that are enabling a new kind of design pattern. We refer to it as the lakehouse. And the idea is: what if you could actually do BI directly on your data lake? And what if you could do your reporting directly on your data lake, and you could do your data science and your machine learning straight up on the data lake?

Martin: I would love to tease apart a few things that have led us here. There’s very clearly a large existing data warehouse market around BI and analytics, typified by people using SQL on structured data.

It seems like the ML and AI use case is a little bit different than the analytics use case. The analytics use case is normally human beings looking at dashboards and making decisions, whereas in the ML/AI use case, you’re creating these models and those models are actually put into production and are part of the product. They’re doing pricing, they’re doing fraud detection, they’re doing underwriting, etc.

The analytics market is an existing buying behavior and an existing customer. ML/AI is an emerging market.