A New Constructive Method to Optimize Neural Network Architecture and Generalization

by   Hou Muzhou, et al.

In this paper, after analyzing the reasons of poor generalization and overfitting in neural networks, we consider some noise data as a singular value of a continuous function - jump discontinuity point. The continuous part can be approximated with the simplest neural networks, which have good generalization performance and optimal network architecture, by traditional algorithms such as constructive algorithm for feed-forward neural networks with incremental training, BP algorithm, ELM algorithm, various constructive algorithm, RBF approximation and SVM. At the same time, we will construct RBF neural networks to fit the singular value with every error in, and we prove that a function with jumping discontinuity points can be approximated by the simplest neural networks with a decay RBF neural networks in by each error, and a function with jumping discontinuity point can be constructively approximated by a decay RBF neural networks in by each error and the constructive part have no generalization influence to the whole machine learning system which will optimize neural network architecture and generalization performance, reduce the overfitting phenomenon by avoid fitting the noisy data.


page 1

page 2

page 3

page 4


A Constructive Approach for One-Shot Training of Neural Networks Using Hypercube-Based Topological Coverings

In this paper we presented a novel constructive approach for training de...

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

In this paper we propose to study generalization of neural networks on s...

Empirical study of extreme overfitting points of neural networks

In this paper we propose a method of obtaining points of extreme overfit...

An Empirical Study of Incremental Learning in Neural Network with Noisy Training Set

The notion of incremental learning is to train an ANN algorithm in stage...

A Constructive Prediction of the Generalization Error Across Scales

The dependency of the generalization error of neural networks on model a...

Approximate Probabilistic Neural Networks with Gated Threshold Logic

Probabilistic Neural Network (PNN) is a feed-forward artificial neural n...

An Interpretable Constructive Algorithm for Incremental Random Weight Neural Networks and Its Application

Incremental random weight neural networks (IRWNNs) have gained attention...

Please sign up or login with your details

Forgot password? Click here to reset