Soft-Deep Boltzmann Machines

by   Taichi Kiwaki, et al.

We present a layered Boltzmann machine (BM) that can better exploit the advantages of a distributed representation. It is widely believed that deep BMs (DBMs) have far greater representational power than its shallow counterpart, restricted Boltzmann machines (RBMs). However, this expectation on the supremacy of DBMs over RBMs has not ever been validated in a theoretical fashion. In this paper, we provide both theoretical and empirical evidences that the representational power of DBMs can be actually rather limited in taking advantages of distributed representations. We propose an approximate measure for the representational power of a BM regarding to the efficiency of a distributed representation. With this measure, we show a surprising fact that DBMs can make inefficient use of distributed representations. Based on these observations, we propose an alternative BM architecture, which we dub soft-deep BMs (sDBMs). We show that sDBMs can more efficiently exploit the distributed representations in terms of the measure. Experiments demonstrate that sDBMs outperform several state-of-the-art models, including DBMs, in generative tasks on binarized MNIST and Caltech-101 silhouettes.


page 7

page 8

page 18

page 19


Boltzmann machines as two-dimensional tensor networks

Restricted Boltzmann machines (RBM) and deep Boltzmann machines (DBM) ar...

What is (missing or wrong) in the scene? A Hybrid Deep Boltzmann Machine For Contextualized Scene Modeling

Scene models allow robots to reason about what is in the scene, what els...

Modeling Documents with Deep Boltzmann Machines

We introduce a Deep Boltzmann Machine model suitable for modeling and ex...

Multi-layered Discriminative Restricted Boltzmann Machine with Untrained Probabilistic Layer

An extreme learning machine (ELM) is a three-layered feed-forward neural...

Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?

It is generally assumed that the brain uses something akin to sparse dis...

Learning Feature Hierarchies with Centered Deep Boltzmann Machines

Deep Boltzmann machines are in principle powerful models for extracting ...

Please sign up or login with your details

Forgot password? Click here to reset