Validating Data in the Data Lake: Best Practices

Can you trust the data in your data lake? Many companies are guilty of dumping data into the data lake without a strategy for keeping track of what’s being ingested. This leads to a murky, swampy repository. If you don’t have transparency into your lake so that you can feel confident using the data, what’s the point of deploying a data lake in the first place?

You know Hadoop is a different animal than the data warehouse, requiring distinct technologies and skill sets. Unlike relational databases, Hadoop is little help when it comes to quality control. Without incorporating additional tools into your data lake architecture you have no way to apply metadata to your data as it is ingested. You can’t automate metadata management so that you can scale to the volume and velocity of big data. You also have no way to customize rules for different data types from different sources.

How do you design your data lake architecture and what functionality do you need to validate the data in your data lake? From our years of experience deploying data lakes for leading companies across heavily regulated industries like financial services and healthcare, we’ve developed some best practices to help companies clean up and derive more value from their data lakes.

Managing data ingestion requires thinking about where the data should land in your lake and where it goes after it’s ingested, in line with your data lifecycle management strategy. We recommend creating zones in the file system of your data lake, dedicated for specific uses; namely,  “transient,” “raw,” “trusted” and “refined” zones. By building a rule-based architecture tied to the metadata that’s applied upon ingestion, you can automate validating the data as you move it from zone to zone. You also may want to incorporate a discovery sandbox “zone,” moving trusted data there for wrangling, discovery and exploratory analysis.