I Have a Data Warehouse, Do I Need a Data Lake Too?
When building your data and analytics program, you must decide whether you need a data warehouse, a data lake, or both. Understanding the difference is the first step.
Anyone building a data and analytics program has run across the terms data warehouse and data lake. You might even have a data warehouse already, serving up dashboards and scorecards to your users. If you are like me, you might question how your data warehouse is different from a data lake and whether both need to be part of your overall analytics strategy.
The answer is these two constructs have different roles in the organization. A data warehouse and a data lake differ in data structure, user base, and governance model. Once you understand these differences, it becomes much easier to evaluate whether you need just one or both in combination.
The first key difference between the data warehouse and the data lake is the structure of the data. The data warehouse has been around since the 1970s and has been described by Bill Inmon as a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management’s decisions. What does each of these parts mean?
Subject-oriented: The data is structured around a specific business subject area (e.g., sales, inventory, or employees). Often these warehouses grow subject by subject until they can answer the questions the business asks. Integrated: As an organization’s data is spread across multiple transactional systems, data is merged and integrated to present a holistic view of the subject area. This entails a data quality process and matching and merging data elements to create cohesion among disparate data. Time-variant: Historical data is maintained in the data warehouse. Changes to the data are maintained to allow reporting to accurately represent historical point-in-time views. An example is keeping past customer addresses to accurately represent sales by location across time and not update those sales as the customer relocates. Non-volatile: Once data is loaded into the data warehouse, it doesn’t change. This creates stability for reporting. Versioning and slowly changing dimensions are used to show changes over time.
A data lake is much different. It is a repository of data in its raw or natural format. This can include relational, semistructured, unstructured, and even binary data. The idea with a data lake is to have less data restructuring and data cleansing to expedite loading this data quickly from multiple sources. This doesn’t mean the data lake is pure chaos; it is structured for optimal loading and for eventual structuring. The goal is to implement structure on data during the read process, merging and munging the raw data into something valuable on an as-needed basis.
A data lake is also structured to keep all data — both data that is being used and data that could be used in the future. The extensive nature of the data set in the data lake creates an optimal source from which to build out the data warehouse now and in the future. As new subject areas are identified as desirable in the data warehouse, a data lake already has the data necessary to do the initial build and the ongoing incremental loads.
Because a data warehouse is specifically formulated to answer subject-oriented questions, it has the highest usability level.
