A novel approach to solving analytical data challenges, Data mesh is rapidly gaining ground—and for good reason.
“Data mesh and its principles are an increasingly hot topic,” says Chris Atkinson, Global Partner CTO, Snowflake. What makes it particularly interesting, Chris says, is that “it’s a combination of a cultural shift in organizations and a technical shift in how we look at analytical data, how we approach its usage and distribution.”
As businesses move towards a data-driven culture, to drive competitive advantage and gain more insightful decision-making, Chris says the reality is that despite a significant rise in investment in data, particularly analytics and Big Data, tangible returns have actually fallen.
He continues, “For years, businesses took a monolithic approach to analytical data, with centralized IT services. This led to the rise of ETL and ELT, and with the domains (business lines) sittingoutsidethe structure. Cycle time is the killer of all processes, and this set-up created huge cycle time.”
The solution – moving analytical functions into business domains – itself led to analytical silos, creating islands of information with interoperability and other challenges to solve, with a continued need for ETL and ELT across the different pipelines and silos: “Spaghetti, in short, in the face of ever-increasing data-driven demands from the business, with more data generated each day, and more access required.”
Data Mesh is a way to solve this. A cultural solution combined with a technical solution, Chris Atkinson says it’s founded on four pillars:
In a real-life example of data mesh, the DataOps.live platform is helping data product teams in the world’s largest biotech company, Roche, to orchestrate and benefit from next-generation analytics on a self-service data and analytics infrastructure comprising Snowflake and other tools.
Roche Diagnostics had previously pursued a number of initiatives to unlock the value of its data, but each had its limitations. “I chose to pursue a forward-looking data stack and, on the architecture side, a novel data mesh approach,” says Omar Khawaja, Global Head of Business Intelligence. “DataOps.live is exploiting every functionality Snowflake provides, bringing us the true DataOps practices we need. It enables our teams to create the data products we require, using all governance best practices like code check-in and check-outs, and allows multiple data engineers to work concurrently in the same team without creating a bottleneck or interfering with each other.”
Paul Rankin,Head of Data Platforms at Roche adds:“Federating this approach across hundreds of developers in 20+ data product teams was the challenge… DataOps.live enables us to pull all this together in terms of orchestration, deployment, release management — and to do it at scale. It’s a complete game changer.”
“Many of the principles of creating data products for data mesh architecture have analogies in software development,” says Guy Adams, CTO, DataOps.live. “We can use those experiences to accelerate adoption. At the same time, we have already implemented data mesh with real-life customers that are now live. Some are huge, with many petabytes of data and one customer has more than 30 different domains.”
1. Data Product Boundaries: the demarcation between product and user must be clear. “As with software, users of data products don’t need visibility of the ‘data internals’ but they do need to know their product will not suddenly change, and is backwards compatible in any case.” Guy says. “If a new version is needed, that has to interoperate with the previous version for a while.”
2. Don’t release too fast but deploy frequently:“I know of one organization which decided what it thought it needed from a particular data product, built it, released version 1, and within a week the users said ‘this isn’t what we needed, it’s not fit for purpose’.
