6 best practices for metadata storage and management
Organizations generate metadata so quickly that it’s turning into today’s big data challenge. However, the best way and location to store all this information is not always clear.
Much depends on how organizations manage and use the metadata and whether they separate it from its primary data. Many organizations move their metadata into a central repository to improve operations and better use the information. As a result, the issue of metadata storage is front and center.
What is metadata storage and why is it important? Simply put, metadata is data about data. It can include a variety of information about a data file, depending on the type of file and the type of storage. For example, a file’s metadata might include the following: date and time it was created; GPS coordinates of where it was created; Although applications often generate metadata automatically, organizations can add it manually and customize it. As data volumes have grown, so too has the important role that metadata plays in managing and optimizing data. Metadata makes it easier to index, find, sort and categorize data. It helps to better understand the data through advanced analytics. Metadata also helps to improve data quality, optimize data management, simplify storage administration and facilitate greater productivity, all of which can lead to more efficient operations and lower costs. Metadata storage must meet the needs of the larger metadata management strategy by providing a safe and efficient system for hosting the data. To realize these benefits, however, an organization must put into place a system for effective metadata storage and management. Metadata storage must meet the needs of the larger metadata management strategy by providing a safe and efficient system for hosting the data. Without a carefully planned and implemented storage system, performance can suffer, data resources can be difficult to find, and metadata can even get lost. The storage system must ensure that the metadata is continuously available.
Best practices for metadata storage and management For a metadata management system that relies on a central repository separate from the source data, storage teams need to consider several factors, including how they will implement and distribute the platform. 1. Don’t go it alone The team’s metadata storage strategy should be part of the organization’s larger metadata management strategy, which in turn should be part of the organization’s larger data governance policy. Effective metadata management requires participation across the entire organization, including the team responsible for metadata storage. Be involved in defining metadata objectives and adopting standards. In this way, the team can bring an important perspective to the discussion and start working on the ground floor. 2. Look at the big picture Although the storage team focuses primarily on metadata storage, have a good grasp of the underlying infrastructure and technology that will support the metadata effort. Understand what components the organization will deploy, how those components fit together and how the metadata will move between those components before finally landing in storage. The team needs to know these details: how the organization will implement the catalog; which database system to use; additional information (other than metadata) that needs storage in the catalog; whether there will be an abstraction layer; which third-party management tools to use; and any other information about the various systems that could affect storage. 3. Think big and then think bigger The storage team should have a clear sense of how the metadata management platform will scale out to meet user demand. Consider such issues as how many sites to deploy and how many storage nodes per site. Understand how the organization will distribute the metadata. Have a clear sense of what it will take to scale storage systems up or out to meet future demand.
