How to put an effective metadata strategy in place

The proliferation of analytics has made data a valuable commodity. According to Gartner, organizations digging for value in data will spend an estimated $18.3 billion on BI and analytics tools in 2017. Yet recent studies show that on average, less than half of an organization’s structured data is actively used in making decisions—and less than 1 percent of unstructured data is analyzed or used at all.

Here’s a big part of the problem: despite the hefty spend, many organizations don’t have effective strategies and tools for leveraging metadata—the invisible data that defines data. That’s a mistake.

Metadata is not new. But it has become more important as data volumes and varieties have grown. Metadata forms the “keys to the kingdom” of data: it tells users what data exists, what it means, where it comes from and whether it’s accurate. And metadata defines the “lingua franca” for interoperation, enabling the increasingly disparate world of data sources and tools to understand how and where to access data and work together.

In recent years, changes in best practices and the software platforms to support them bring higher risks and rewards. Given the emergence of data lakes, self-service analytics, open source and cloud platforms, metadata becomes even more necessary to handle the increase in variety and volume of data sets. And as the value of metadata grows, there’s a sensible worry in the air that proprietary metadata solutions could lock up the keys to the data that is so important to a well governed analytics organization.

Since the beginning, metadata has been good for “defense”: ensuring that data usage stays within guidelines and protects bottom-line business processes. For an overall data strategy, that means incorporating governance into policy-driven IT practices, engineered in advance of data loading and usage.

Enterprises need to know: Is critical data secure and trustworthy? Is there an “audit trail” of data lineage that shows when and how data is generated, used and changed? Is the data correct—standardized in its representation and content?

Defense is about prevention. As such, a defensive metadata approach needs to be designed and deployed in advance, providing a constrained framework for top-down control of data representations.

What’s been changing in recent years is the growing value of playing “offense” with data, bringing new perspectives to metadata strategy. The modern data-driven enterprise empowers creative analysts to generate top-line value by aggressively harnessing any data they can get. Business-driven value generation is the signature feature behind the “big data” and “data science” movements, representing a key competitive advantage (or threat!) across nearly every business sector.

Offense is about creating opportunities to score. It requires agility and flexibility on the field: each line of business in an organization should be promiscuous in finding useful data, quick to innovate with process, tools and techniques, and free to change behavior as they learn from experience and continuously improve.

A Freakonomics study from a few years back argues that, in football at least, you need a balanced strategy of offense and defense. I agree in this context too: successful organizations need to allow both defensive metadata engineering in advance, and offensive metadata capture and curation on use.

Embracing new data practices goes hand-in-hand with embracing new platforms and tools, and the advent of open source and the cloud have accelerated those transitions. As data migrates to these new platforms and practices, there are open questions about the nature of the next generation of metadata solutions. On that front, here are a few simple principles to keep in mind in selecting metadata solutions and rolling them out for usage.

The first principle: require openness. Organizations are aggressively moving data out of platforms from legacy software vendors with lock-in strategies.