An Introduction to Deep Learning using nolearn
One of the most well known problems in machine learning regards how to categorize handwritten numbers automatically. Basically, the idea is that you have 10 different digits (0-9) and you want a computer to be able to correctly identify each number. This would come in handy at the post office, along with many other applications such as identifying address numbers in images.
Different machine learning experts have worked on this problem for several years trying a variety of approaches. Ultimately, however, the best algorithm for the task of image classification would be an algorithm that can fit features that aren’t easily described. This is where neural networks truly shine.
Deep learning has become quite the trendy subject recently. Here is an example. Essentially, researchers are trying to use Deep Learning to categorize images better than a human could, something humans are typically better at doing naturally than a computer.
If you want an idea as to how Deep Learning is being used (and how it works) listening to a talk from Dr. Andrew Ng is always a good idea!
Basically, the key part of deep learning is finding features that can be used for classification from UNLABELED training examples. More data = a more accurate model, most of the time. Labeling the myriad of images available on the web one at a time into categories would be very difficult and time consuming (although Amazon paid people money to do this through their Mechnical Turk program). Instead, just find features from your unlabeled images (such as certain curves, color gradients, or other features that humans would have a difficult time describing). Then, use these features to help categorize your images. It greatly improves the accuracy because now you have more data to work with!
As a way of practicing this, we are going to use some Deep Learning on the MNIST dataset of numbers to see if we can get a low error rate. I couldn’t decide which neural network library to use at first, since scikit-learn doesn’t really have any. I became inspired by this post here from Dr. Adrian Rosebrock. He used the library nolearn, which we will also be using.
I just found out while writing this that the dbn class of nolearn is being removed entirely for the new version 0.6 and that the author suggests switching to lasagne. If you are going to follow along with the IPython Notebook, make sure you have a version prior to nolearn 0.6.
Just to get a starting idea of where we are beginning, let’s try a very basic run of a deep belief network using nolearn. Because image processing can be fairly memory intensive, I am including the key steps of processing the data and wrapping them inside a function.
Our function will load the MNIST data, transform it to a 0 to 1 scale for all of the pixel intensity values, and finally split the data into train and test sets. We need to scale the data between 0 to 1 so that the neural network will perform more efficiently.
Now, let’s run our function to split the data and scale it.
Given that we now have train/test data, let’s try making a simple deep belief network and get a baseline error rate to start with.
Notice that our version is not using cudamat. It would run a lot faster if we had a nice GPU to use, but since I am on a laptop that doesn’t have a nice one, we are going to run this using the slower, CPU way.
We are using the example settings shown in the documentation just as a benchmark. So what do all of these settings mean?
Now it is time to train our deep belief network. This could take a little bit . . .
After about 3 and a half minutes, our network is ready! Let’s see how accurate it is on the test data, just to get an idea of where we are starting from.
So, our starting accuracy is pretty close to 98% at 97.95%.
