What can we learn from gradients?

by   Jia Qian, et al.

Recent work (<cit.>) has shown that it is possible to reconstruct the input (image) from the gradient of a neural network. In this paper, our aim is to better understand the limits to reconstruction and to speed up image reconstruction by imposing prior image information and improved initialization. Firstly, we show that for the non-linear neural network, gradient-based reconstruction approximates to solving a high-dimension linear equations for both fully-connected neural network and convolutional neural network. Exploring the theoretical limits of input reconstruction, we show that a fully-connected neural network with a one hidden node is enough to reconstruct a single input image, regardless of the number of nodes in the output layer. Then we generalize this result to a gradient averaged over mini-batches of size B. In this case, the full mini-batch can be reconstructed in a fully-connected network if the number of hidden units exceeds B. For a convolutional neural network, the required number of filters in the first convolutional layer again is decided by the batch size B, however, in this case, input width d and the width after filter d^' also play the role h=(d/d^')^2BC, where C is channel number of input. Finally, we validate and underpin our theoretical analysis on bio-medical data (fMRI, ECG signals, and cell images) and on benchmark data (MNIST, CIFAR100, and face images).


page 7

page 8

page 10

page 11

page 14


An Equivalence of Fully Connected Layer and Convolutional Layer

This article demonstrates that convolutional operation can be converted ...

Width Provably Matters in Optimization for Deep Linear Neural Networks

We prove that for an L-layer fully-connected linear neural network, if t...

A case where a spindly two-layer linear network whips any neural network with a fully connected input layer

It was conjectured that any neural network of any structure and arbitrar...

Adaptive Measurement Network for CS Image Reconstruction

Conventional compressive sensing (CS) reconstruction is very slow for it...

Understanding the Initial Condensation of Convolutional Neural Networks

Previous research has shown that fully-connected networks with small ini...

A Two-step-training Deep Learning Framework for Real-time Computational Imaging without Physics Priors

Deep learning (DL) is a powerful tool in computational imaging for many ...

Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks

In an attempt to better understand structural benefits and generalizatio...

Please sign up or login with your details

Forgot password? Click here to reset