A Tour of Recurrent Neural Network Algorithms for Deep Learning
Recurrent neural networks, or RNNs, are a type of artificial neural network that add additional weights to the network to create cycles in the network graph in an effort to maintain an internal state.
The promise of adding state to neural networks is that they will be able to explicitly learn and exploit context in sequence prediction problems, such as problems with an order or temporal component.
In this post, you are going take a tour of recurrent neural networks used for deep learning.
After reading this post, you will know:
Note, we’re not going to cover every possible recurrent neural network. Instead, we will focus on recurrent neural networks used for deep learning (LSTMs, GRUs and NTMs) and the context needed to understand them.
We will start off by setting the scene for the field of recurrent neural networks.
Next, we will take a closer look at LSTMs, GRUs, and NTM used for deep learning.
We will then spend some time on advanced topics related to using RNNs for deep learning.
Popular belief suggests that recurrence imparts a memory to the network topology.
A better way to consider this is the training set contains examples with a set of inputs for the current training example. This is “conventional, e.g. a traditional multilayered Perceptron.
But the training example is supplemented with a set of inputs from the previous example. This is “unconventional,” e.g. a recurrent neural network.
As with all feed forward network paradigms, the issues are how to connect the input layer to the output layer, include feedback activations, and then train the construct to converge.
Let’s now take a tour of the different types of recurrent neural networks, starting with very simple conceptions.
The layered topology of a multilayer Perceptron is preserved, but every element has a weighted connection to every other element in the architecture and has a single feedback connection to itself.
Not all connections are trained and the extreme non-linearity of the error derivatives means conventional Backpropagation will not work, and so Backpropagation Through Time approaches or Stochastic Gradient Descent is employed.
Recursion promotes branching in hierarchical feature spaces and the resulting network architecture mimics this as training proceeds.
Training is achieved with Gradient Descent by sub-gradient methods.
This is described in detail in R. Socher, et al., Parsing Natural Scenes and Natural Language with Recursive Neural Networks, 2011.
Schmidhuber reported a very deep learner, first in 1991, that was able to perform credit assignment over hundreds of neural layers by unsupervised pre-training for a hierarchy of RNNs.
Each RNN is trained unsupervised to predict the next input. Then only inputs generating an error are fed forward, conveying new information to the next RNN in the hierarchy, which then processes at a slower, self-organizing time scale.
It was shown that no information is lost, just compressed. The RNN stack is a “deep generative model” of the data. The data can be reconstructed from the compressed form.
See J. Schmidhuber, et al., Deep Learning in Neural Networks: An Overview, 2014.
Backpropagation fails as the calculation of extremity of non-linear derivatives increases as the error is propagated backwards through large topologies, making credit assignment difficult, if not impossible.
With conventional Back-Propagation Through Time (BPTT) or Real Time Recurrent Learning (RTTL), error signals flowing backward in time tend to either explode or vanish.
The temporal evolution of the back-propagated error exponentially depends on the size of the weights. Weight explosion may lead to oscillating weights, while in vanishing causes learning to bridge long time lags and takes a prohibitive amount of time, or does not work at all.
