Data Mesh: The Answer to the Data Warehouse Hypocrisy

Note: I start this piece with some technical background that has nothing to do with the data mesh, and is only relevant to data warehousing insofar as it explains how the parallel database systems that often underlie data warehousing solutions usually work. The main thesis of this article does not begin until the fourth paragraph. But the first three paragraphs serve as important background behind this thesis.

There is one rule, pretty much the only rule, that determines success in scaling data systems: avoid coordination whenever possible. This one rule neatly summarizes my entire career until this point — the quarter century I’ve spent teaching, innovating, and building scalable data systems. At the end of the day it all comes down to one thing — avoiding coordination.

From a technical perspective, it is obvious why avoiding coordination is so important. No CPU processor can single-handedly handle the resource-intensive workloads that are required in modern data systems. So a scalable system necessarily requires multiple processors, often on the order of thousands or more, typically distributed across multiple independent servers with their own memory and often their own stable storage. When each processor can run completely independently of the other processors, you automatically get linear scalability. If you double the processors, you double the workload the system is able to handle. 

The only thing that gets in the way of linear scalability is coordination. If one processor needs anything from another processor, whether it be data, metadata, lock permission, scheduling permission, or anything at all such that it cannot proceed until it receives what it needs from the other processor, that processor is prevented from being able to continuously make progress, and this introduces scalability bottlenecks. Every system that I’ve been involved in building or designing  — from scalable transaction systems such as Calvin/Fauna, H-Store/VoltDB, Bohm, Orthrus, and SLOG to scalable data analytics systems such as C-Store/Vertica, HadoopDB/Hadapt, and Borealis, the primary innovations behind these systems involved making each processor as independent as possible. 

All of this is why the concept of a “data warehouse” is one of the most hypocritical ideas on the face of the planet. Modern data warehouses are typically built using cutting edge scalable software using the architectural principles that have emerged from my lab along with the many other research labs world-wide that are innovating in the area of scalable systems. Tremendous care is taken to achieve near linear scalability of the software, allowing potentially thousands of machines to operate in parallel with minimal communication and coordination. The improvement in the scalability of these systems and the overall progress that has been made in the past few decades has been stunning, dizzying, and inspiring. 

And yet we go and tell businesses to deploy these super scalable systems inside an organizational structure that is the very antithesis of scalability: the data warehouse. Anybody who has been involved in the deployment of a non-trivial enterprise-wide data warehouse knows that the endeavour is filled with coordination. Organizationally, it is a centralized behemoth, a single source of truth. Centralization and parallelization are antonyms. Scalability requires independent units working in parallel, while centralization introduces coordination, resistance, and inertia. 

Getting data into a data warehouse typically requires a great deal of coordination between those in charge of the source system data, those in charge of data governance, data quality, master data management, data integration, those in charge of the data platform, devops, and the data engineers or data scientists that are driving the incorporation of this new data. It is routine to experience delays of multiple-months or longer to get data included in the data warehouse.