PolyNeuron: Automatic Neuron Discovery via Learned Polyharmonic Spline Activations

11/10/2018
by   Andrew Hryniowski, et al.
0

Automated deep neural network architecture design has received a significant amount of recent attention. However, this attention has not been equally shared by one of the fundamental building blocks of a deep neural network, the neurons. In this study, we propose PolyNeuron, a novel automatic neuron discovery approach based on learned polyharmonic spline activations. More specifically, PolyNeuron revolves around learning polyharmonic splines, characterized by a set of control points, that represent the activation functions of the neurons in a deep neural network. A relaxed variant of PolyNeuron, which we term PolyNeuron-R, loosens the constraints imposed by PolyNeuron to reduce the computational complexity for discovering the neuron activation functions in an automated manner. Experiments show both PolyNeuron and PolyNeuron-R lead to networks that have improved or comparable performance on multiple network architectures (LeNet-5 and ResNet-20) using different datasets (MNIST and CIFAR10). As such, automatic neuron discovery approaches such as PolyNeuron is a worthy direction to explore.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/01/2018

q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators

We propose a new generic type of stochastic neurons, called q-neurons, t...
research
02/24/2017

Activation Ensembles for Deep Neural Networks

Many activation functions have been proposed in the past, but selecting ...
research
02/26/2018

A representer theorem for deep neural networks

We propose to optimize the activation functions of a deep neural network...
research
11/21/2019

DeepLABNet: End-to-end Learning of Deep Radial Basis Networks with Fully Learnable Basis Functions

From fully connected neural networks to convolutional neural networks, t...
research
02/15/2021

And/or trade-off in artificial neurons: impact on adversarial robustness

Since its discovery in 2013, the phenomenon of adversarial examples has ...
research
06/29/2022

Automatic Synthesis of Neurons for Recurrent Neural Nets

We present a new class of neurons, ARNs, which give a cross entropy on t...
research
07/03/2023

First Steps towards a Runtime Analysis of Neuroevolution

We consider a simple setting in neuroevolution where an evolutionary alg...

Please sign up or login with your details

Forgot password? Click here to reset