Demystifying Data Lake Architecture
According to Gartner, 80% of successful CDOs will have value creation or revenue generation as their Number 1 priority through 2021.
To create the maximum value out the organization’s data landscape, traditional decision support system architecture are no longer adequate. New architectural patterns need to be developed to harness the power of data. To fully capture the value of using big data, organizations need to have flexible data architectures and able to extract maximum value from their data ecosystem.
Data Lake concept has been around for some time now. However, I have seen organizations struggle to understand the concept as many of them are still boxed in the older paradigm of Enterprise Data Warehouses.
In this article, I will deep-dive into conceptual constructs of Data Lake Architecture pattern and layout an architecture pattern.
Let us start with the known first.
Traditional Data Warehouse (DWH) Architecture:
Traditional Enterprise DWH architecture pattern has been used for many years. There are data sources, data is extracted, transformed and loaded (ETL) and on the way, we do some kind of structure creation, cleansing etc. We predefine the data model in EDW (dimensional model or 3NF model) and then create departmental data marts for reporting, OLAP cubes for slicing and dicing and self-service BI.
This pattern is quite ubiquitous and has served us well for a long time now.
However, there are some inherent challenges in this pattern that can’t scale in the era of Big Data. Let us look at few of them:
Firstly, the philosophy with which we work is that we need to understand the data first. What is the source system structure, what kind of data it holds, what is the cardinality, how should be the model it based on the business requirements, are there any anomalies in data so on and so forth. This is a tedious and complex work. I used to spend at least 2–3 months in the requirement analysis and data analysis phase. The EDW projects span for a few months to a few years. And this is all based on the assumption that the business knows the requirements.
We also have to make choices and compromises on which data to store and which data to discard. A lot of time is spent upfront on deciding what to bring in, how to bring in, how to store, how to transform etc. Lesser time is spent on actually performing data discovery, uncovering patterns, or creating a new hypothesis for business value add.
Definition of Data:
Let us now discuss briefly how the definition of data has changed. The 4 Vs of big data are now very well known. Volume, velocity, variety, and veracity. Let me put some context to these things:
Data volumes have exploded in since the iPhone revolution. There are 6 billion smartphones and nearly 1PB of data is created every year.
Data is not just at rest. There are streaming data, IoT enabled connected devices. A plethora of data emanating from multiple fronts.
It is also about the variety of data. Video feeds, photographs all are data points now which demand to be analyzed and exploited.
With the explosion of data also comes the challenge of data quality. Which one should be trusted and which one should not be is a bigger challenge in the big data world.
In short, the definition of analysable data has changed. It is not just structured corporate data now but all kinds of data. The challenge is to mash them up together and make sense out of them.
Moore’s Law:
Since 2000 there has been tremendous changes in the processing capabilities, storage, and the corresponding cost structure. It has been subjected to what we call as Moore’s Law.Key points:
Processing capabilities have increased by around 10,000 times since 2000. This implies that the ability to analyze more data efficiently has increased.
The cost of storage has also come down quite considerably. Since 2000, the cost of storage has come down over 1000 times.
