Minimal Viable Prediction: Is Big Data Dead?

Big data is everywhere. Organizations are being advised to hoard it and to do everything they can to derive actionable insights from it. This article will argue that this approach puts the cart before the horse.

First, let’s start with a revelation. Despite the hype and despite everything you’ve been told about big data as a necessary precursor to achieving valuable business insights, it has by and large failed to deliver on its promises.

All the data in the world means nothing if you can’t make sense of it. And that’s the hard part. Even Gartner predicts that through 2017, 60% of big data projects will fail to go beyond piloting and experimentation and ultimately will be abandoned.

Part of the reason for this failure is that organizations rush to accumulate and analyze as much data as possible, all at once, and without much of a plan. This usually ends terribly. It’s not unusual anymore for organizations to house multiple petabytes of data. These compulsive data hoarders pathologically store any data they can get their hands on, thinking that the more data they have, the better off they will be, the more intelligent they will be, and the higher the quality and number of insights they can glean from it.

But starting big can lead to an overwhelmed team, not to mention huge costs. These organizations won’t be able to see the insights for the data, so to speak. With more data and more capabilities to rationalize data across silos, it can feel like there are more opportunities than there are resources to exploit them.

The problem is exacerbated by the fact that traditional approaches often fall short. Heavy duty enterprise business intelligence (BI) tools, central data warehouses, etc. designed to analyze big datasets take years and cost millions to implement (something your BI service provider won’t tell you at the outset).

These heavy data warehouses can pay off. But when they don’t, the cost of failure is high. In a survey of more than 500 executives by McKinsey Analytics, 86% of respondents reported that their organizations were only somewhat effective at meeting the goals they set for their data and analytics initiatives.

The moral of the story? Hoarding big data is not a prerequisite for smart insights.

The truth is, for many organizations, starting small is a much more fruitful (and lower-cost) exercise than bringing in heavyweight tools and consultants. Time and again, we have seen some of the most valuable business insights derived from surprisingly small datasets. It’s time to focus not on big data but on the right data at the right time and find a way to ask the right questions of that data.

It’s an approach to data analytics that we replicated from our best practices in software product development. Focus on delivering an MVP as fast as you can, and iterate from there. In this case, MVP does not stand for minimal viable product, but minimal viable prediction.

The MVP approach challenges traditional perceptions and established norms of big data and data analytics. The notion of having to assemble data at any cost is an extreme one. Instead of eating the entire elephant at once (your mouth is too small and you’ll choke), we recommend a low-risk approach to data analytics that a successful start-up might take. It is grounded in best-in-class software development methodologies. Here’s how it breaks down:

Instead of assembling all your data, disregard the noise and assemble only the data that correlates with your number one problem.

Once you’ve established your primary problem, work backwards from there, identifying other related data, and follow the data breadcrumbs that lead to actionable outcomes. For example, you may start with understanding and predicting customer complaints — something that can have a very real impact on your revenue and profits. So, you assemble complaints data. Your next data breadcrumb might be customer service representative data, a potential reason for complaints.