Driving Faster Insights with a Data Mesh
A Data Mesh approach removes the bottlenecks that come with the traditional data warehousing and data lake models, and in doing so, allows companies to drive faster insights.
The data warehousing model has owned the data management space for years. In this centralized model, data is typically passed from a number of operational systems to a single analytical system, with data engineers and analysts then trying to understand and derive value from that data. This model creates a bottleneck between the data producers and the analysts since we need a third function that is responsible for actually migrating the data from operational systems to the analytical system. This bottleneck results in an “air gap” between the data producers and the analysts, meaning that the analysts cannot be as “close” to the data and its original context, and this bottleneck becomes a key scalability constraint.
The data warehouse was traditionally designed to enable and support an organization with business intelligence through analytics, reports, and dashboards. This approach is predicated on consolidating data assets to create a “single source of truth.”
One of the reasons that the Data Lake concept was born was to overcome some of the challenges associated with Data Warehouses. In this approach, data is again brought from multiple sources to a single location for the purposes of analysis, but in this approach, the data is not modeled to form a single source of truth when it is bought together. Instead, the data is effectively modeled when it is exploited by the analysts, so now each analyst can determine their own single source of truth. This is a very flexible model for the analyst but suffers from a lack of governance around the data and thus the decision-making. However, the bigger concern is that it has not removed the need for a central team for getting the data to the Data Lake in the first place.
To some degree, the Data Warehouse and Data Lake are at opposite ends of the spectrum when it comes to defining a single source of truth but are very similar when it comes to the centralization of ownership of the process of getting data from the operational systems to the analytic systems. This centralization of ownership slows down the organization’s ability to respond to changing business needs.
See also: Data Lakes, Time-Series Data, and Industrial Analytics
Zhamak Dehghani, a principal technology consultant at ThoughtWorks, introduced Data Mesh as “a sociotechnical approach to share, access and manage analytical data in complex and large-scale environments – within or across organizations.
