There are so many buzzwords these days regarding data management. Data lakes, data warehouses, and databases – what are they? In this article, we’ll walk through them and cover the definitions, the key differences, and what we see for the future.
If you want full, in-depth information, you can read our article called, “What’s a Data Lake?” But here we can tell you, “A data lake is a place to store your structured and unstructured data, as well as a method for organizing large volumes of highly diverse data from diverse sources.”
The data lake tends to ingest data very quickly and prepare it later, on the fly, as people access it.
A data warehouse collects data from various sources, whether internal or external, and optimizes the data for retrieval for business purposes. The data is primarily structured, often from relational databases, but it can be unstructured too.
Primarily, the data warehouse is designed to gather business insights and allows businesses to integrate their data, manage it, and analyze it at many levels.
Essentially, a database is an organized collection of data. Databases are classified by the way they store this data. Early databases were flat and limited to simple rows and columns. Today, the popular databases are:
And, of course, there are other terms such as data mart and data swamp, which we’ll cover very quickly so you can sound like a data expert.
Enterprise Data Warehouse (EDW): This is a data warehouse that serves the entire enterprise.
Data Mart: A data mart is used by individual departments or groups and is intentionally limited in scope because it looks at what users need right now versus the data that already exists.
Data Swamp: When your data lake gets messy and is unmanageable, it becomes a data swamp.
Data lakes, data warehouses and databases are all designed to store data. So why are there different ways to store data, and what’s significant about them? In this section, we’ll cover the significant differences, with each definition building on the last.
Databases came about first, rising in the 1950s with the relational database becoming popular in the 1980s.
Databases are really set up to monitor and update real-time structured data, and they usually have only the most recent data available.
But the data warehouse is a model to support the flow of data from operational systems to decision systems. What this means, essentially, is that businesses were finding that their data was coming in from multiple places—and they needed a different place to analyze it all. Hence the growth of the data warehouse.
For example, let’s say you have a rewards card with a grocery chain. The database might hold your most recent purchases, with a goal to analyze current shopper trends.
