Why you should use open data to hone your machine learning models

One of the big reasons we created our Data for Everyone initiative is that there simply aren’t a ton of great open datasets out there for small businesses, startups, and academics to do work on. Sure, there are plenty of small, toy-sized datasets but those simply aren’t big enough to create algorithms that anyone can trust. In fact, our founder Lukas wrote as much in this post:

Remember when Netflix ran a contest to beat it’s own movie rating algorithm? Tens of thousands of solutions were submitted, all based off a single dataset of about 100,000,000 rows and Netflix eventually awarded the million dollar prize to a team of data scientists who beat the company‘s algorithm by more than 10%. Even now, a half decade since the prize was awarded, that Netflix data set is constantly used in computer science research — over 3,000 papers mention it. And almost all of the papers that mention it were written after the contest ended. It’s not that movie data is so important to computer science research — there just aren’t many good quality datasets available. The contest wasn’t the important thing — releasing the data was the real value to the world.

We believe that open data should become the new open source; we think shared clean, enriched data is one of the key ingredients to real innovation.

That’s why we were incredibly happy to find out what the folks over at MonkeyLearn are doing with some of the sets we’ve shared on our Data for Everyone library. MonkeyLearn is a machine learning application that extracts and classifies information from text. They let you upload custom training sets and create your own machine learning algorithm that’s appropriate for your specific use-case. After all, different organizations have different vernacular and a one-size-fits-all solution works like one-size-fits-all t-shirt: it doesn’t really fit anyone all that well.

Since MonkeyLearn extracts and classifies text, they do a lot of sentiment analysis work. As they write in their post “sentiment analysis is damn hard” and “it’s one of the most complex machine learning tasks out there.” That’s true. Just because people inherently understand opinion and feeling in text better than a computer can doesn’t mean we always agree. There are competing metrics out there for how much people concur about the sentiment of a particular piece of text, but commonly, you hear about 70% and, again, different industries have different words, slang, subcultures, in-jokes, etc. that don’t translate to other fields. To put it another way: a basketball “MVP” is a most valuable player; a tech “MVP” is a minimum viable product. Those aren’t even close to the same thing.

That’s why the folks MonkeyLearn are so keen on custom sentiment analysis. Creating an algorithm based on a specific use is always going to better than a generic, out-of-the-box natural language processor (NLP).

But don’t take our word for it. Here’s how MonkeyLearn used CrowdFlower to not only set a baseline for their sentiment model, but used free, open data from our Data for Everyone library to create industry-specific algorithm that outperformed every other machine learning solution.

First, they collected about 2,000 tweets about brands, celebrities, movies, and tech. They ran a CrowdFlower job to label the tweets (meaning our pool of contributors tagged each individual tweet as positive, negative, or neutral) and compared those judgements to their sentiment algorithm, as well as a few other models out there.

That 65.