Enhancing approximation abilities of neural networks by training derivatives
share
Method for increasing precision of feedforward networks is presented. With the aid of it they can serve as a better tool for describing smooth functions. Namely, it is shown that when training uses derivatives of target function up to the fourth order, approximation can be nearly machine precise. It is demonstrated in a number of cases: 2D function approximation, training autoencoder to compress 3D spiral into 1D, and solving 2D boundary value problem for Poisson equation with nonlinear source. In the first case cost function in addition to squared difference between output and target contains squared differences between their derivatives with respect to input variables. Training autoencoder is similar, but differentiation is done with respect to parameter that generates the spiral. Supplied with derivatives up to the fourth the method is found to be 30-200 times more accurate than regular training provided networks are of sufficient size and depth. Solving PDE is more practical since higher derivatives are not calculated beforehand, but information about them is extracted from the equation itself. Classical approach is to put perceptron in place of unknown function, choose the cost as squared residual and to minimize it with respect to weights. This would ensure that equation holds within some margin of error. Additional terms used in cost function are squared derivatives of the residual with respect to independent variables. Supplied with terms up to the second order the method is found to be 5 times more accurate. Efficient GPU version of algorithm is proposed.
READ FULL TEXT