Data fabric or data mesh? Don’t move ahead without an active metadata strategy
“After me, the deluge,” is attributed to French King Louis XV or his mistress, Madame de Pompadour, in reference to signs of the approaching Revolution.
The virtually unlimited amount of data, processing capacity and tools to leverage it are a modern deluge, without a doubt. The challenge today is not the volume of data, it’s making sense of it, at scale, continuously.
No organization today is immune from the push for some form of digital transformation. The late Peter Drucker famously said, “The computer may have aggravated management’s degenerative tendency to focus inward.” That was almost twenty years ago and is almost certainly not true today. However, it illustrates how information systems have changed, and how quickly.
It is no longer solely sufficient to thrash through your internal record-keeping systems for insight, and it is very likely that you already do your analytics in multiple locations, platforms, clusters and with very different kinds of data. In addition, more of your staff are engaged in analytics due to better software tools, and more will continue to be. But you need help.
The sudden surge in computing capacity drove demand for more data and analytics but also led to a temporary structural shortage of professional data scientists and statisticians. Exacerbating this insufficiency were productivity problems associated with the work, known as the 80/20 problem: 80% of the time spent managing data and only 20% doing the quantitative investigation. This claim first appeared at least eight years ago, and it still endures. There is no clear, rigorous evidence that this 80% measure is accurate. Depending on the circumstances, it varies widely by organization, application, and the skill and tools applied. However, it is impossible to deny that sourcing data for analytical and data science uses is a significant effort, regardless of the percentage cited.
Clearly, better tools are needed to improve highly skilled and compensated professionals’ productivity (and job satisfaction). Even those performing analytics in organizations in more traditional ways can benefit from an intelligent and integrated product taking them from data discovery and profiling to navigating the data with a dynamic, semantically rich data catalog and applying it in unique and productive ways.
The cadence of technology innovation surpasses most organizations’ ability to implement each new or improved technique before the next one arrives. Data management can never be a pure, complete process. It requires trade-offs, picking the issues that make the most sense, have the most significant centrality to the organization’s strategy (ies), provide the most protection against danger, and ensure the organization can be as effective as possible. Rich metadata-driven catalogs are emerging as the solution.
It would be a mistake to assume that a static catalog, even one replete with valuable information, would be an adequate solution.
