A developmental approach for training deep belief networks

by   Matteo Zambra, et al.

Deep belief networks (DBNs) are stochastic neural networks that can extract rich internal representations of the environment from the sensory data. DBNs had a catalytic effect in triggering the deep learning revolution, demonstrating for the very first time the feasibility of unsupervised learning in networks with many layers of hidden neurons. These hierarchical architectures incorporate plausible biological and cognitive properties, making them particularly appealing as computational models of human perception and cognition. However, learning in DBNs is usually carried out in a greedy, layer-wise fashion, which does not allow to simulate the holistic maturation of cortical circuits and prevents from modeling cognitive development. Here we present iDBN, an iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the model. We evaluate the proposed iterative algorithm on two different sets of visual stimuli, measuring the generative capabilities of the learned model and its potential to support supervised downstream tasks. We also track network development in terms of graph theoretical properties and investigate the potential extension of iDBN to continual learning scenarios. DBNs trained using our iterative approach achieve a final performance comparable to that of the greedy counterparts, at the same time allowing to accurately analyze the gradual development of internal representations in the deep network and the progressive improvement in task performance. Our work paves the way to the use of iDBN for modeling neurocognitive development.


page 7

page 14

page 18


Robust Continual Learning through a Comprehensively Progressive Bayesian Neural Network

This work proposes a comprehensively progressive Bayesian neural network...

Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing

Training deep neural networks results in strong learned representations ...

Is Continual Learning Truly Learning Representations Continually?

Continual learning (CL) aims to learn from sequentially arriving tasks w...

Deep learning systems as complex networks

Thanks to the availability of large scale digital datasets and massive a...

Gamma Belief Networks

To infer multilayer deep representations of high-dimensional discrete an...

Design Exploration of Hybrid CMOS-OxRAM Deep Generative Architectures

Deep Learning and its applications have gained tremendous interest recen...

Please sign up or login with your details

Forgot password? Click here to reset