Data or Algorithms – Which is More Important?

Summary:  Which is more important, the data or the algorithms?  This chicken and egg question led me to realize that it’s the data, and specifically the way we store and process the data that has dominated data science over the last 10 years.  And it all leads back to Hadoop.

Recently I was challenged to speak on the role of data in data science.  This almost sounds like a chicken and egg problem.  How can you have one without the other?  But as I reflected on how to explain this it also struck me that almost everything in the press today is about advances in algorithms.  That’s mostly deep learning and reinforcement learning which are driving our chatbots, image apps, and self-driving cars.

So if you are fairly new to data science, say within the last five or six years you may have missed the fact that it is and was the data, or more specifically how we store and process the data that was the single most important factor in the explosion of data science over the last decade.  In fact there was a single innovation that enabled data lakes, recommenders, IoT, natural language processing, image and video recognition, AI, and reinforcement learning.

Essentially all of these areas of major innovation can be tracked back to the single enabler, NoSQL Hadoop.

It was in 2006 that Doug Cutting and his team took the proprietary work done at Google to the Apache Institute and created open source Hadoop.

Most of you will recognize that this was also the birth of the era of Big Data, because Hadoop for the first time gave us a reasonable way to store, retrieve, and analyze anything.  The addition of unstructured and semi-structured data like text, speech, image, and video created the possibilities of AI that we have today.  It also let us store volumes of ordinary data like web logs or big transactional files that were previously simply too messy to store.

What you may not know, and I heard Doug Cutting himself quote at this last spring’s Strata Conference in San Jose is that the addition of unstructured and semi-structured data are not the most important feature of Hadoop.  The most important feature is that it allowed many ordinary computers to function as a single computer.  This was the birth of Massive Parallel Processing (MPP).  If it hadn’t been for MPP the hardware we have today would never have evolved and today’s data science simply would not and could not exist.

It’s interesting to track the impact that this has had on each of the major data science innovations over the last decade:

I have personally been practicing in predictive analytics since 2001.  As valuable as that discipline was becoming to any major company with a large B2C market, we were restricted to basically numerical data.

As we move through this history I’ll use this graphic to help locate the impact of the ‘data’ versus the innovation it enables.  On the vertical axis we have the domains of structured through unstructured data.  On the horizontal axis, a description of whether that data science technique delivers very specific insights or just more directional guidance.

For the most part, in predictive modeling we were restricted to what we could extract from RDBMS systems like a BI warehouse, or with much more effort from transactional systems.  A few of our algorithms like decision trees could directly handle standardized alpha fields like state abbreviations, but pretty much everything had to be converted to numeric.

Predictive models on the other hand deliver business insights that are extremely specific about consumer behavior or the future value of a target variable.  Generally, predictive models continue to deliver accurate predictions in the range of 70% to 90% accuracy about questions like who will buy or what the spot price of oil will be next month.

One of the first applications of our new found compute power and flexibility was Data Lakes.