Deep Diving the Data Lake – Automatically Determining What’s In There
As your data lake grows larger and your user group more diverse you will need these tools that automatically catalog data and control access to your information. They are a huge benefit and only enhance the spirit of free exploration of data for new value.
Data Lakes have been around almost as long as we’ve had Big Data. That means this year could be their 10 birthday. And they are still a valuable approach to storing data and making it widely available for exploration without the pain and suffering associated with putting it in your EDW.
The big idea was to trade:
The basis for all RDBMS, for:
This is what we know as ‘late schema’ or ‘schema on read’. And the value continues to be that we don’t burden IT with the need to create new schemas, tables, and data, and in return we get our data on demand, not weeks or months from now.
We’re not saying bad things about IT. We’re saying that the growth in demand for stored and extracted data by data scientists, analysts, and line of business users is growing so fast that IT can’t keep up. Also, that EDWs based on RDBMS simply require too much work to modify for our new mindset, “let’s see what’s in there”.
Early data lakes were the meat-and-potatoes of data scientists working on new projects. They could take data from transactional systems, from EDWs, and from new Big Data sources like text and streams, create their own data lake, and go to town.
Because these early users were largely data scientists and had good SQL and coding skills as well as some decent data engineering chops they could do this in a self-contained way that served their immediate set of projects. If only one or a few data scientists were using it, they controlled what went in, how it was processed, and what it was used for.
But as data lakes evolved into more robust systems and more and more users wanted this capability, these older narrow-use instances didn’t meet the need of the much larger group of folks who wanted larger and larger varieties of data. To democratize data you needed something larger than these siloed data puddles.
The number of vendors providing turnkey, easy to establish and easy to use data lakes has exploded. And the use of data lakes has exploded as well because the same dynamics about the time-delays and costs still exist, while the number of users clamoring for data has grown.
It’s more common now, and certainly more efficient to have a single large data lake than any number of smaller data puddles. Data lakes are now much more likely to be created and supported by IT specialists and data engineers. That’s a good thing.
Another positive aspect of creating centralized data lakes is that it tends to support a new company-wide ethos to save everything and analyze it later, even if we don’t yet know what the value will be.
Storage is cheap and a remarkable amount of potentially valuable data is being discarded all the time. Take weblogs for example or CSR recordings or transcripts. All sorts of interesting and value-enhancing models are being built from these sources.
But here’s the rub. The more data you put in there the harder it is for any given user to know where to find it, how valid it is, how clean or dirty it might be, whether it duplicates other data also in the lake, or any of a host of other reasons to slow down and be concerned as you undertake your analytic or modeling project. One source says accurately “finding data in data lake is like shopping in a flea market”.
To further complicate the matter, a wider group of users means a much wider set of skills.
