Why deep learning is an essential tool for developers
The field of artificial intelligence has shown tremendous progress in the past decade. But there’s more to AI than chess-playing robots. Mat Leonard, the Head of Udacity’s School of AI, explains how the history of deep learning is the history of a programming revolution. Are you ready for Software 2.0?
The last few years have seen amazing progress in the field of artificial intelligence. AlphaGo beat the world’s Go grandmasters, a feat thought impossible. The next version, AlphaZero, became the world’s best chess player in four hours. Cars are driving themselves, smartphones can identify skin cancer as well as trained dermatologists, and we’re on the verge of human-level universal translation. All of this is the result of not just machine learning, but actually a subset technology of machine learning called deep learning. But what is deep learning, what can it do, and how did we get to this point?
When people talk about AI these days, they are actually talking about deep learning. Artificial intelligence is a broad field that includes pathfinding algorithms such as A*, logic and planning algorithms, and machine learning. The field of machine learning consists of various algorithms with internal parameters found from example data through optimization. Deep learning is a branch of machine learning utilizing “deep” neural networks, that is, artificial neural networks with dozens of layers and millions of parameters. The recent advances in AI such as speech recognition, realistic image generation, and AlphaZero are all based on deep learning models.
Artificial neural networks have existed since the 1950s, which were known as perceptrons at that time. The perception was an algorithm that roughly approximates the way neurons operate. An individual neuron (or “unit”) summed up its input values, each value multiplied by some connection strength or weight. The sum of those values and weights was then passed through an activation function to get the output of the neuron. The neurons could be combined into layers with multiple neurons in each layer, using the output of neurons in one layer as the input to neurons in the next layer. The weights would be set such that the network performs some specified behavior. Most of the time setting the weights by hand is practically impossible. Instead, the network was “trained” using example data. That is, the input data is labeled and the weights are adjusted such that the network is able to reproduce the correct labels from the data.
After the initial excitement, researchers were blocked because they couldn’t train neural networks with more than two layers, restricting the ability of the networks to perform complex behaviors. Two decades later in the 1980s, a solution was found in the backpropagation algorithm, which allowed information to flow through the network from the output layers back to the input layers. Suddenly, researchers could train deeper neural networks with multiple layers. However, the process of training was computationally expensive and while there were some successes, neural networks weren’t seen as better alternatives to other machine learning algorithms.
In the 2012 ImageNet competition, Alex Krizhevsky and Ilya Sutskever from Geoffrey Hinton’s lab trained a deep neural network with 60 million parameters on two GPUs for a week. The goal of the competition was to identify objects in images with the lowest error rate, using a dataset of 1.2 million images as training examples. Their algorithm dominated the field with an error rate of 15%, beating the next best attempt by ten percentage points. Afterwards, deep neural networks became the only choice for computer vision problems. This combination of massive neural networks, trained on gigantic datasets and using GPUs to increase computational efficiency, is the basis of deep learning and all the amazing breakthroughs we’re seeing in AI.
Deep learning has created a new paradigm for software development.
