Hadoop is Not Failing, it is the Future of Data

The author disagrees with a previous KDnuggets post on “Why Hadoop is Failing” and argues that the Darwinian Open Source Ecosystem ensures Hadoop is a robust and mature technology platform .

“Madam..What use is a new-born baby?”’ –Michael Faraday – (Apocryphal quote) when asked about the utility of electricity a new invention in the 1800s…

As my readers are aware I have been heavily involved in the Big Data space for the last two years. This time has been an amazing and transformative personal experience as I have been relentlessly traveling the globe advising global banking leaders across all continents.

Thus, it should come as no surprise that the recent KDnuggets article somewhat provocatively titled – Hadoop is Failing – Why, by George Hill – managed get me disagreeing right from the get go .

The author, though well meaning from what I can tell, bases the article on several unfounded assumptions. Before we delve into those, let us consider the following background.

The onset of Digital Architectures in enterprise businesses implies the ability to drive continuous online interactions with global consumers/customers/clients or patients. The goal is not just provide engaging visualization but also to personalize services customers care about – while working across multiple channels/modes of interaction. Mobile applications first began forcing the need for enterprise applications  to support multiple channels of interaction with their consumers. For example Banking now requires an ability to engage consumers in a seamless experience across an average of four to five channels – Mobile, eBanking, Call Center, Kiosk etc. Healthcare is a close second where caregivers expect patient, medication & disease data at their fingertips with a few finger swipes on an iPad app.Big Data technology evolved to overcome the limitations of existing data approaches (RDBMS & EDW)  to keep up data architecture & analysis challenges inherent in the Digital application stack.

Big Data is backed by the open source community with most Hadoop ecosystem technology (25+ projects) incubated, developed and maintained in the Apache ecosystem. The Open Source community is inherently Darwinian in nature. Its focus on code quality, industry adoption, a concrete roadmap and committers means that If a project lacks  then it is for sure headed for the graveyard.  Put another way, there can be no stragglers in this ecosystem.

Let us now consider the chief assumptions made by the author in the above article.

The best part of my job is working with multiple customers daily on their business initiatives and figuring out how to apply technology to solving these complex challenges. I can attest that adoption in the largest enterprise is anything but stagnating. While my view is certainly anecdotal and confined to the four walls of one company, adoption is indeed skyrocketing at verticals like Banking, Telecom, Manufacturing & Insurance. The early corporate movers working with the leading vendors, have more or less figured out the kinks in the technology as applied to their business challenges. The adoption patterns are maturing and they are realizing massive business value from it. A leading vendor, Hortonworks, moved to $100 million annual revenues quicker than any other tech startup – which is testament to the potential of this space. Cloudera just went public. All this growth has been accompanied by somewhat declining revenues & stock prices at leading EDW vendors. I forecast that the first Big Data ‘startup’ to $1 billion in revenue will happen over the next five-seven years, at a somewhat faster rate compared to the revered open source pioneer Red Hat. At a minimumHadoop projects cut tens of millions of dollars from costly and inflexible enterprise data warehouse projects. Nearly every large organization has begun deploying Hadoop as an Enterprise Landing Zone (ELZ) to augment an EDW.