The hidden horse power driving Machine Learning models

Machine Learning is becoming the only real available method to perform many modern computational tasks in near real time. Machine Vision, speech recognition and natural language processing have all proved difficult to crack with out ML techniques.

When it comes to hardware, the tasks themselves do not need a great deal of computational power; but training the machine does – not to mention an awful lot of data. In the machine learning world, the more data you have the more accurate your ML model can be. Of course the more data you have the longer the training process will take.

Take for example the Imagenet 2012 dataset, a set of test images that can be used for evaluating machine learning methods. This data set is a mere 138GB that I’m told will take one day to download to my machine. Once I have the data, it can be fed into the training phase of a machine learning algorithm – typically an iterative process where the model is trained with data, tested, the parameters tweaked and then the model is trained again.

Each iteration is termed an “epoch” in machine learning terms. For an idea of how long this can take, let’s to the Tensorflow program I built for learning movie recommendations. This will typically learn in 100 epochs fairly good recommendations for movies. Development was carried out on a dataset of 100,000 ratings, and this can realistically carried out on a MacBook Pro in less than an hour. Once you move up to the full 24 million ratings, however, the training time per epoch moves to more than 140 seconds, which will run for around nine days on the MacBook Pro. This really isn’t viable.

Something needs to be done. Maybe we could move this problem into the cloud and let the big boys with their big machines take over. The problem is moving your data into the cloud. For universities and the likes of Google, this isn’t really a problem, providing you’ve got access to end to end fast networks. Universities in Britain are all connected over the Janet network, whose backbone runs at 100Gbps, more than enough to shift large datasets around. Google, of course, has its own dark net, but what if we want to move data out of our walled garden and onto a public cloud ML system?

This was just the problem we faced a few years back at Dundee University when trying to use Microsoft’s Azure a to process Mass Spectrometer data. These files were fairly big – a few gigabytes in size – but we were hoping to process lots of them in near real time. Sadly, we just couldn’t get the data into the cloud fast enough.

This is why Amazon released Snowball, essentially a box of hard disks delivered to your door and that you fill with data (at 10Gbps) and return to be loaded into AWS. It is remarkably cheap around $200 per job with 10 days to fill the disks with your data, but it is not real-time.

It is for this reason that companies are starting to offer hardware that can be situated close to the data production (in terms of network speed) for machine learning. These appliances employ GPUs to speed up the maths needed for machine learning. Essentially the mathematics performed to produce the fast action in a video game is similar to that done in machine learning (and other fields); you just want to do lots of sums in parallel on fairly small chucks of data.

There are a number of manufacturers of GPU cards – Nvidia, AMD and Intel’s Xeon – but it is only really Nvidia that has grasped the ML nettle and made it easy to do deep learning on its cards.

So what sort of speed-up can you get? I used an Nvidia Titan X and set it up to run the Film recommendation engine with the large dataset. The training per epoch fell from 140 seconds to around 11, making training this large dataset a realistic proposition.