Data lake and data warehouse – know the difference
Over the past few years, you may have heard someone somewhere drop the term “data lake.” The concept has increasingly gained traction as data volumes have increased exponentially, streaming data has taken off, and unstructured data has continued to dwarf its structured counterpart.
But what is a data lake anyway? Is it just marketing hype? And, generally speaking, how does it differ from the traditional data warehouse? Odds are that at some point in your career you’ve come across a data warehouse, a tool that’s become synonymous with extract, transform and load (ETL) processes. At a high level, data warehouses store vast amounts of structured data in highly regimented ways. They require that a rigid, predefined schema exists before loading the data. (It’s almost always a star or snowflake schema.) Put differently, the schema in a data warehouse is defined “on write.”
ETL processes dutifully kick out error reports, generate logs, and send errant records to exception files and tables to be addressed at later dates. Because of this rigidity and the ways in which they work, data warehouses support partial or incremental ETL. In other words (and depending on the severity of the issue), an organization can load or reload portions of its data warehouse when something goes wrong. Organizations typically populate data warehouses periodically. Generally speaking, data refreshes via regular cycles – say every morning at 3 a.m. when employees aren’t likely to be accessing the data and downstream systems.
Employees arrive at work the next day with freshly squeezed data. To be sure, the data stored in traditional data warehouses remains valuable today. Still, organizations and their leaders need to begin rethinking contemporary data integration. Consider the Internet of Things and the analytics it makes possible. Sensors on vehicles, farm equipment, wearables, thermostats and even crops result in massive amounts of data that stream continuously. It’s a good bet that even an industrial-strength data warehouse will struggle with these new streams of data.
The rise of the data lake Against this backdrop, we’ve seen the rise in popularity of the data lake. Make no mistake: It’s not a synonym for data warehouses or data marts. Yes, all these entities store data, but the data lake is fundamentally different in the following regard. As David Loshin writes, “The idea of the data lake is to provide a resting place for raw data in its native format until it’s needed.” Data lies dormant unless and until someone or something needs it. The specific data types and sources they need. How much they need. When they need it.
The types of analytics that they need to derive. Are all of these possible in a data warehouse? Probably not. And even if they were possible, achieving them in a period of time that business users would find acceptable is unlikely – especially in today’s rapidly changing environments. Beyond that, one particular schema almost certainly will not fit every business need. To wit, the data may ultimately arrive in a way that renders it virtually useless for the employee’s evolving purposes.
For this very reason, a data lake schema is defined “on read.” Put differently, a data lake still requires a schema. However, that schema is not predefined. It’s ad hoc. Data is applied to a plan or schema as users pull it out of a stored location – not as it goes in. Data lakes keep data in its unaltered (natural) state; it doesn’t define requirements unless and until users query the data. When used correctly, data lakes offer business and technical users the ability to query smaller, more relevant and more flexible data sets. As a result, query times can drop to a fraction of what they would have been in a data mart, data warehouse or relational database.
The increased flexibility of the data lake The data lake emphasizes the flexibility and availability of data.
