How to Guide: Architecture Patterns to Consider When Designing an Enterprise Data Lake
Let’s start with the standard definition of a data lake:
“A data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed…and a question:“Why should you care?
In a large enterprise, perhaps the most powerful impact of a data lake is the enablement of innovation. We have seen many multi-billion dollar organizations struggling to establish a culture of data-driven insight and innovation. They get bogged down by the structural silos that isolate departmental or divisionally-divided data stores, and which are mirrored by massive organizational politics around data owner-ship. While far from trivial to implement, an enterprise data lake provides the necessary foundation to clear away the enterprise-wide data access problem at its roots. The door to previously unavailable exploratory analysis and data mining opens up, enabling completely new possibilities.
In today’s dynamic business environment, new data consumption requirements and use cases emerge extremely rapidly. By the time a requirements document is prepared to reflect requested changes to data stores or schemas, users have often moved on to a different or even contradictory set of schema changes. In contrast, the entire philosophy of a data lake revolves around being ready for an unknown use case. When the source data is in one central lake, with no single controlling structure or schema embedded within it, supporting a new additional use case can be much more straightforward.
What is the average time between a request made to IT for a report and eventual delivery of a robust working report in your organization? In far too many cases, the answer is measured in weeks or even months. With a properly designed data lake and well-trained business community, one can truly enable self-service Business Intelligence. Allow the business people access to what ever slice of the data they need, letting them develop the reports that they want, using any of a wide range of tools. IT becomes the custodian of the infrastructure and data on the cloud, while business takes responsibility for exploring and mining it.
The foundation of any data lake design and implementation is physical storage. The core storage layer is used for the primary data assets. Typically it will contain raw and/or lightly processed data. The key considerations when evaluating technologies for cloud-based data lake storage are the following principles and requirements:
Because an enterprise data lake is usually intended to be the centralized data store for an entire division or the company at large, it must be capable of significant scaling without running into fixed arbitrary capacity limits.
As a primary repository of critical enterprise data, a very high durability of the core storage layer allows for excellent data robustness without resorting to extreme high-availability designs.
One of the primary design considerations of a data lake is the capability to store data of all types in a single repository.
The ability to apply schema upon read, as needed for each consumption purpose, can only be accomplished if the underlying core storage layer does not dictate a fixed schema.
The most significant philosophical and practical advantage of cloud-based data lakes as compared to “legacy” big data storage on Hadoop is the ability to decouple storage from compute, enabling independent scaling of each.
Given the requirements, object-based stores have become the de facto choice for core data lake storage. AWS, Google and Azure all offer object storage technologies.
The point of the core storage is to centralize data of all types, with little to no schema structure imposed upon it. However, a data lake will typically have additional “layers” on top of the core storage.
