A model of semantic completion in generative episodic memory

by   Zahra Fayyaz, et al.

Many different studies have suggested that episodic memory is a generative process, but most computational models adopt a storage view. In this work, we propose a computational model for generative episodic memory. It is based on the central hypothesis that the hippocampus stores and retrieves selected aspects of an episode as a memory trace, which is necessarily incomplete. At recall, the neocortex reasonably fills in the missing information based on general semantic information in a process we call semantic completion. As episodes we use images of digits (MNIST) augmented by different backgrounds representing context. Our model is based on a VQ-VAE which generates a compressed latent representation in form of an index matrix, which still has some spatial resolution. We assume that attention selects some part of the index matrix while others are discarded, this then represents the gist of the episode and is stored as a memory trace. At recall the missing parts are filled in by a PixelCNN, modeling semantic completion, and the completed index matrix is then decoded into a full image by the VQ-VAE. The model is able to complete missing parts of a memory trace in a semantically plausible way up to the point where it can generate plausible images from scratch. Due to the combinatorics in the index matrix, the model generalizes well to images not trained on. Compression as well as semantic completion contribute to a strong reduction in memory requirements and robustness to noise. Finally we also model an episodic memory experiment and can reproduce that semantically congruent contexts are always recalled better than incongruent ones, high attention levels improve memory accuracy in both cases, and contexts that are not remembered correctly are more often remembered semantically congruently than completely wrong.


page 4

page 5

page 7

page 8

page 10

page 11


DualGenerator: Information Interaction-based Generative Network for Point Cloud Completion

Point cloud completion estimates complete shapes from incomplete point c...

CompNVS: Novel View Synthesis with Scene Completion

We introduce a scalable framework for novel view synthesis from RGB-D im...

Point-PC: Point Cloud Completion Guided by Prior Knowledge via Causal Inference

Point cloud completion aims to recover raw point clouds captured by scan...

The Tensor Memory Hypothesis

We discuss memory models which are based on tensor decompositions using ...

β-VAEs can retain label information even at high compression

In this paper, we investigate the degree to which the encoding of a β-VA...

Grounded and Controllable Image Completion by Incorporating Lexical Semantics

In this paper, we present an approach, namely Lexical Semantic Image Com...

Pluralistic Image Completion

Most image completion methods produce only one result for each masked in...

Please sign up or login with your details

Forgot password? Click here to reset