Memorization in Deep Neural Networks: Does the Loss Function matter?

07/21/2021
by   Deep Patel, et al.
0

Deep Neural Networks, often owing to the overparameterization, are shown to be capable of exactly memorizing even randomly labelled data. Empirical studies have also shown that none of the standard regularization techniques mitigate such overfitting. We investigate whether the choice of the loss function can affect this memorization. We empirically show, with benchmark data sets MNIST and CIFAR-10, that a symmetric loss function, as opposed to either cross-entropy or squared error loss, results in significant improvement in the ability of the network to resist such overfitting. We then provide a formal definition for robustness to memorization and provide a theoretical explanation as to why the symmetric losses provide this robustness. Our results clearly bring out the role loss functions alone can play in this phenomenon of memorization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/26/2022

Hybridised Loss Functions for Improved Neural Network Generalisation

Loss functions play an important role in the training of artificial neur...
research
06/14/2019

Empirical study of extreme overfitting points of neural networks

In this paper we propose a method of obtaining points of extreme overfit...
research
10/02/2020

Effective Regularization Through Loss-Function Metalearning

Loss-function metalearning can be used to discover novel, customized los...
research
01/30/2023

Online Loss Function Learning

Loss function learning is a new meta-learning paradigm that aims to auto...
research
03/03/2023

Certified Robust Neural Networks: Generalization and Corruption Resistance

Adversarial training aims to reduce the problematic susceptibility of mo...
research
06/07/2023

Loss Functions for Behavioral Game Theory

Behavioral game theorists all use experimental data to evaluate predicti...
research
11/30/2021

LossPlot: A Better Way to Visualize Loss Landscapes

Investigations into the loss landscapes of deep neural networks are ofte...

Please sign up or login with your details

Forgot password? Click here to reset