Managed Data Lakes Deliver Exceptional Value and Accessibility
Many enterprises are feeling let down by the promise of Big Data and data lakes because they’re not seeing the return on their investment that they expected. Their original approach to data lake management wasn’t scalable; therefore, a new approach is required to help them derive significant value from their data. Specifically, these enterprises need a way to effectively manage and govern all of their data and make it readily available for use for various levels of users — beyond data scientists.
We think this new approach involves managed data lakes. In fact, we predict the future enterprise data ecosystem will have a managed data lake at its core. The data lake will be fed by multiple structured data sources, real-time data streams, and unstructured data. All of the data will be stored in this central repository, in the cloud, on-premise, or in some combination where it can be transformed, cleaned, and manipulated by data scientists, data analysts, and general business users. Then, prepared datasets can be fed back into the data warehouse for business intelligence or to other visualization tools for data science, data discovery, analytics, predictive modeling, and reporting.
Many early adopters used a Hadoop data lake as a relatively inexpensive storage solution and dumped data into it without much of a plan, expecting they would figure it out when they needed to use the data. The problem is that there is just too much data of various quality in too many formats. To keep track of it all and enable data governance, data must be managed upon ingestion. Data management is achieved by layering on a data management platform to your data lake that applies metadata and defines, tracks, and logs all steps of what data is ingested into the data lake. A data lake management platform is what provides the essential data visibility, reliability, security, and privacy controls, provides an understanding of data quality, and can allow broader access to data by multiple users.
