Over the last decade, the increased use of unstructured and alternative data, and the ascendance of the cloud, has posed an overt challenge to the relational database model based on structured inputs and batch transmission. Transition from the classic SQL on Hadoop database toward the unstructured, real time data hub or lake has come with costs, however.
These costs are best understood within the context of a four stage data lifecycle comprising capture, transformation, extraction and delivery. Challenges related to capture and transformation revolve around inconsistent data quality and manual intervention requirements. Extraction and delivery processes, on the other hand, tend to founder on the expectations of the end user or business unit.
End user dissatisfaction with existing business intelligence processes and the need for more timely delivery of pricing, operational and analytical insight has spurred a generalized reliance on workarounds, which may include the unauthorized input and removal of data. The knock-on effects of these workarounds have in turn fostered desire for an end-to-end, all embracing data solution.
The process of eliminating data marts (and the superstructure repositories that contain them) presupposes massive resource deployment as well as a philosophical about-face. Resource deployment extends beyond data clean up and transition from legacy architecture to the investment required to build and support the data lake structure. The philosophical about-face relates to the fundamental purpose of data within an organization.
In this new world data management is not a top-down process designed for a single use case. Rather, improved data management should enable the creation of a so-called golden source, in which universally accepted and accessible data is summoned by business partners according to their needs.
Look Before You Leap
The data lake that is able to store and process data in its native or raw format represents the latest iteration of this vision, but not the end state.
