5 algorithms to train a neural network
The procedure used to carry out the learning process in a neural network is called the training algorithm. There are many different training algorithms, with different characteristics and performance.
The learning problem in neural networks is formulated in terms of the minimization of a loss function, f. This function is in general, composed of an error and a regularization terms. The error term evaluates how a neural network fits the data set. On the other hand, the regularization term is used to prevent overfitting, by controlling the effective complexity of the neural network.
The loss function depends on the adaptative parameters (biases and synaptic weights) in the neural network. We can conveniently group them together into a single n-dimensional weight vector w. The picture below represents the loss function f(w).
As we can see in the previous picture, the point w* is minima of the loss function. At any point A, we can calculate the first and second derivatives of the loss function. The first derivatives are grouped in the gradient vector, whose elements can be written as Similarly, the second derivatives of the loss function can be grouped in the Hessian matrix,
The problem of minimizing continuous and differentiable functions of many variables has been widely studied. Many of the conventional approaches to this problem are directly applicable to that of training neural networks.
Although the loss function depends on many parameters, one-dimensional optimization methods are of great importance here. Indeed, they are very often used in the training process of a neural network.
Certainly, many training algorithms first compute a training direction d and then a training rate η that minimizes the loss in that direction, f(η). The next picture illustrates this one-dimensional function.
The points η and η define an interval that contains the minimum of f, η*.
In this regard, one-dimensional optimization methods search for the minimum of a given one-dimensional function. Some of the algorithms which are widely used are the golden section method and Brent’s method. Both reduce the bracket of a minimum until the distance between the two outer points in the bracket is less than a defined tolerance.
The learning problem for neural networks is formulated as searching of a parameter vector w at which the loss function f takes a minimum value. The necessary condition states that if the neural network is at a minimum of the loss function, then the gradient is the zero vector.
The loss function is, in general, a non-linear function of the parameters. As a consequence, it is not possible to find closed training algorithms for the minima. Instead, we consider a search through the parameter space consisting of a succession of steps. At each step, the loss will decrease by adjusting the neural network parameters.
In this way, to train a neural network we start with some parameter vector (often chosen at random). Then, we generate a sequence of parameters, so that the loss function is reduced at each iteration of the algorithm. The change of loss between two steps is called the loss decrement. The training algorithm stops when a specified condition, or stopping criterion, is satisfied.
Now, we are going to describe the most important training algorithms for neural networks.
Gradient descent, also known as steepest descent, is the simplest training algorithm. It requires information from the gradient vector, and hence it is a first order method.
Let denote f(w) = f and ᐁf(w) = g. The method begins at a point w and, until a stopping criterion is satisfied, moves from w to w in the training direction d = -g. Therefore, the gradient descent method iterates in the following way:
The parameter η is the training rate. This value can either set to a fixed value or found by one-dimensional optimization along the training direction at each step.
