How NoSQL Fundamentally Changed Machine Learning

Summary:  In just the six or seven short years since the first commercial implementation of a Hadoop NoSQL database Machine Learning has come to mean so much more than it did before.

The question often comes up from folks starting to explore data science, just what is Machine Learning? When I started out it was easy to explain. Machine Learning (ML) was the category of mathematical algorithms like regression, clustering, decision trees, and neural nets used to extract signals from data, aka predictive models. Then came NoSQL and all that changed.

The first commercial NoSQL implementation (not counting Google’s first mover efforts) are credited to Yahoo’s implementation of Hadoop in 2008 to improve their search indexing. The first Hadoop developers conference was mid-2008, and early implementation by Facebook, Twitter, and eBay occurred in 2009. This entire explosion in capability is now barely six or seven years old.

So what exactly has ML become? How do they relate? In an effort to explain this recently, I put the components on this grid.

On the vertical I’ve put the major categories of data along with the NoSQL DB types that most commonly corresponds, and on the horizontal an indication of whether the insights gained from this ML type are specific or simply directional.

These have always required numeric data (OK decision trees can take some text categoricals) and historically they drew on our traditional structured data from transactional systems or a data warehouse. The insight gained was intended to be quite accurate and if you’ll allow me a little artistic leeway, I’d say we expected these to be say 70% to 95% accurate in forecasting specific future values or human behaviors.

In a nod to NoSQL I let this category lap into Semi-Structured data territory where we can now extract specific features using JSON or XML coding to feed our models.

And while I’m not sure that “data lakes” rise to the level of an ML tool, the ability of NoSQL to create blended databases of semi-structured data that can handle all the volume, variety, and velocity of Big Data sources needs to be acknowledged.

Recommenders are those wonderful tools that tell us who to date or what to watch or read. Originally they were the exclusive domain of NoSQL Graph DBs but increasingly they’re being built with NoSQL Columnar DBs like Hbase. I don’t think anyone would argue they are meant to be anything more than directionally correct but their existence has undoubtedly added millions and millions of dollars in guided ecommerce buys and perhaps even some happy marriages.