MapR takes a stab at data governance, in an age of data anarchy
Data governance — the discipline of inventorying and annotating your data sets, determining their accuracy, pedigree and quality and properly securing them — is an important focus area for the industry. In the conventional database world, Enterprise Information Management (including ETL, data quality management and master data management) has addressed these needs for some time. In the data lake world, though, efforts have been far less earnest.
Granted, there are data catalog products, and lineage products. There are various security/access control solutions and there are metadata management systems as well. Cloudera has its Navigator product, and there’s even open source Apache Atlas (incubating), borne of a Hortonworks project called the “Data Governance Initiative.” Some analytics products even have governance features of their own, enticing customers away from bringing in another vendor and platform to handle governance requirements.
On Tuesday, MapR announced a data governance initiative of its own. It is comprised of an interesting architectural approach, a few key partnerships, and a services offering to go with it all. I’ll first detail MapR‘s announced offering, and I’ll conclude with some analysis (aka a rant) on the state of data governance in the Big Data world.
Topic: PreprocessorOn the technical side, MapR has come up an approach that, to me at least, is pretty novel and clever. The company has taken a prescriptive stance here and is advising customers that all data ingestion should go through MapR Event Streams (MapR-ES, formally known just as MapR Streams), the company‘s Kafka API-based publish/subscribe platform for handling event-based data ingest.
The hook, as it were, is this: by configuring a pre-processor on the MapR-ES topic, all data pushed through it can be observed, its discernible metadata captured in a MapR-DB document database, and metadata changes can also recorded there. This allows for metadata cataloging and, if derivative data set creation is managed similarly, and all MapR-ES events are retained, data lineage can be determined comprehensively, just by “playing back” the events.
The partner part So MapR provides the raw infrastructure to get metadata and lineage information. But it doesn’t offer a data catalog facility that would let data lake users search for data sets, tag them, see which of them are certified and see star ratings for them, provided by other users.
That’s where partners and their products come in. Waterline Data and Collibra, each of which offers data catalog and data lineage functionality, are key partners. Cask, whose Data Application Platform (CDAP) provides a unified API over various Big Data components, and specific APIs for metadata inspection and for audit, is a partner as well.
By themselves, each of these products only catalogs what’s entered into them. They work as long as everyone uses them (or codes to them, in the case of CDAP).
