Where Next for Metadata
Metadata finally makes the big time, but how big and for how long? The answers lie in traditional data warehousing principles.
It’s ironic that data lakes — possibly the most poorly managed data stores in the history of IT — have finally, after nearly three decades, focused the industry’s attention on the need for metadata. Ironic but unsurprising. Metadata is prescribed to make data usable and useful to both business and IT, but dumping data flotsam and information jetsam in ungoverned lakes leaves everybody struggling to figure out what’s what and where it might be.
Gartner’s 2019 Metadata Management Solutions Magic Quadrant (also available from many of the listed vendors) reflects the burgeoning growth in interest, with the number of vendors evaluated almost doubling to 17 since the 2016 version. The Leaders quadrant is rather packed: fully a dozen of the contenders are placed there, with Informatica leading the race for the top right corner. Alation, Alex Solutions, ASG, Collibra, Infogix, and Smartlogic form a bunch of six in the middle of the quadrant, while Adaptive, erwin, IBM, Oracle, and SAP tend lower left among the Leaders.
Inclusion and rankings aside, the metadata market is now well populated with vendors tackling different aspects of function and scope. Indeed, Gartner’s Peer Insights lists 32 products from 22 vendors of metadata management solutions. Software vendors clearly see a worthwhile opportunity, but how real is it?
Gartner’s Magic Quadrant helpfully estimates the number of customers for the included vendors. It suggests that customer uptake, while significant, is still slow. Although a few big hitters (such as Informatica, Oracle, and SAP) boast a few thousand clients each, many others cluster around a hundred and below. I counted six to eight thousand in total, which is a fraction of data warehouse and data lake customers worldwide.
Data warehouse experts like myself have been promoting the need for metadata since the early 1990s. One might imagine a tsunami of pent-up demand being released now that a plethora of powerful products and simple solutions have arrived on the market. Even allowing for the possibility that many data warehouses are ticking over in maintenance mode and thus closed to new investment in metadata function, there has been a significant migration of data warehouses to lakes in recent years. Why is the uptake of metadata products not considerably faster?
I have long argued that metadata is a wholly inadequate word to describe what it is and what it does. Metadata is far more than “data about data” — a typically simplistic definition. That is but one limited subset of what is needed, which is to provide business and IT users with sufficient background and context to enable them to create and use data and IT systems with complete confidence. Furthermore, because data is only a subset of information, what’s really needed is “information about information (and data and systems and people and…),” leading to my preferred name: context-setting information or CSI for short.
