data mesh has been steadily gaining popularity as an approach to enterprise data architecture. While it embodies some great ideas, there are a few crucial principles that could easily be lost when attempting to apply the approach in real life. The core idea of data mesh is to organize an enterprise data architecture by logical, business-oriented domains, implemented primarily by distributing responsibilities among the “owners” of each domain. This contrasts with the traditional approach that organizes teams by layers within the technical architecture (ingestion, data management, access, etc.) and is driven primarily by centralized coordination.
As enterprises and their data assets and processes become increasingly complex, establishing a method to manage that complexity makes a lot of sense. And putting as much responsibility as possible into the hands of people most familiar with each domain (business processes, data subjects, etc.) is the best way to match the right expertise and experience to the solution components associated with those domains. Sounds good so far.
But the problem with data mesh – or, rather, the problem with how it has been initially implemented in multiple organizations – is that in a well-meaning effort to reduce complexity and distribute responsibilities, many of the positive aspects of centralization are lost. That is, the baby (highly coordinated, cross-functional capability) is being thrown out with the bathwater (excessive centralization and unmanageable complexity).
To avoid this troubling trend, data leaders must preserve a few important, time-tested principles that remain relevant and require careful attention when following the data mesh philosophy.
I’ve now encountered multiple instances where, in an attempt to follow the data mesh approach, a central data governance organization divided the data landscape into named domains, assigned data owners to each domain to lead planning and implementation for the assigned domain, and essentially said, “Go!” You may be able to guess the results.
Without adequately aligning to business drivers to justify the effort, scope the work across domains, and prioritize elements of the implementation, each data owner prioritized implementation in seemingly random ways, based on the data they considered “important”.
Any data element within each domain was arguably within scope, and any data issue encountered was a candidate for deep analysis and resolution. Without proactive and rigorous coordination, there was no way to ensure the domains could support cross-functional business initiatives on any reasonable schedule anyone could count on.
The result was long and costly projects with limited use of the deployed data – and a lot of frustrated application projects left on their own to gather and manage the data they needed. Any approach to architecture must start with business initiatives; especially funded, cross-functional business initiatives that are, by definition, important to the enterprise.
Here, I’m not referring to “data initiatives” with names like “data marketplace” or “single version of the truth” (as useful as those concepts are in support of business initiatives).
Instead, the drivers should be initiatives like omni-channel customer experience, supply chain optimization, manufacturing automation, and the like. With real business initiatives as drivers, each data owner can commit to delivering the elements within their domain to support the near-term business need.
This approach dramatically narrows the focus of each incremental delivery and provides a clear scoping mechanism across domains. Every element delivered has strategic value while contributing to coherent data at the same time.
Let’s acknowledge that the concept of data warehousing is, at least in part, a response to technical limitations.
