Data Lake Business Model Maturity Index

“Our organization is abuzz with the concept of data lakes!” a customer recently told me. And rightfully so, as the data lake holds the potential to help organizations become more effective at leveraging data and analytics to power their business models. That’s exactly what we propose when we talk about the Big Data Business Model Maturity Index, and helping organizations to exploit the power of predictive, prescriptive, and cognitive (self-learning) analytics to advance up the Business Model maturity index.

But thinking about the data lake as only a technology play is where organizations go wrong. And in fact, thinking of the data lake as only a data repository (something akin to your data warehouse) can create a chasm that hinders the organization’s ability to leverage data and analytics for business value, which hinders an organization’s ability to “monetize” its data by optimizing key operational processes, mitigating compliance and security risks, uncovering new revenue opportunities, and creating a more compelling customer engagement.

From our customer experiences with respect to building out their data lakes, we’d like to share our Data Lake Business Model Maturity Index. This Data Lake Business Model Maturity Index not only shows you where you are today with respect to leveraging your data lake to drive monetization opportunities, but also provides a roadmap for getting from where you are today to where you need to be tomorrow.

Data Lake 1.0:  Getting Familiar with the Technology
Data Lake 1.0 is where organizations are standing up and getting familiar with big data technologies such as Hadoop, HDFS, Hive and HBase. Generally, the goal with these early data lakes was to offload as much data as possible to lower the overall cost of performing analytics. However, organizations are making some big mistakes as they build out their Data Lake 1.0; creating “anti-patterns” or “worst practices” that will ultimately hinder their ability to create a scalable, elastic data platform.
Too much Hadoop. The first anti-pattern is “too much Hadoop” and by that we mean that Hadoop distributions or clusters are all over the enterprise, with duplicated data. The typical Hadoop deployment model for many enterprises starts with a little Hadoop and then expands. Then a second department implements their own Hadoop and expands, and then a third department implements their own Hadoop and expands, and so on.This creates siloes of data, which defeats the purpose of big data analytics. Executives can’t perform analytics across all departments because of the siloed Hadoop deployments. Essentially, organizations have re-created the data warehouse/mart data proliferation problem, just using more modern technology.

Too much governance. The next anti-pattern is “too much governance.” Some organizations take the concept of governance too far by building a data lake with so many restrictions on who can view, access and work on the data, that no one ends up being able to access the lake.

Not enough governance. The opposite of too much governance is “not enough governance,” where organizations lack data stewards, tools, and policies to manage access to the data in the lake. What tends to happen is that there is a tremendous amount of data in the lake that no one really knows what it is for, the quality is low, and eventually the business cannot trust the data.

Inelastic Architecture. The most common anti-pattern is “inelastic” architecture. This pattern occurs because organizations slowly and organically grow their big data environment one server at a time, often buying cheaper servers initially, but eventually adding very expensive servers to keep up with the demands of the business.