Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks

by   Rufin VanRullen, et al.

While objects from different categories can be reliably decoded from fMRI brain response patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a large dataset of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dimensional description of each image. We then presented several thousand face images to human subjects, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Finally, we applied this mapping to novel test images, turning the obtained fMRI patterns into VAE latent codes, and ultimately the codes into face reconstructions. Qualitative and quantitative evaluation of the reconstructions revealed robust pairwise decoding (>95 (PCA decomposition). Furthermore, this brain decoding model can readily be recycled to probe human face perception along many dimensions of interest; for example, the technique allowed for accurate gender classification, and even to decode which face was imagined, rather than seen by the subject.


page 1

page 2

page 4

page 5

page 6

page 7

page 16

page 17


Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

Decoding and reconstructing images from brain imaging data is a research...

Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning

Reconstructing perceived images based on brain signals measured with fun...

FMRI data augmentation via synthesis

We present an empirical evaluation of fMRI data augmentation via synthes...

DCNN-GAN: Reconstructing Realistic Image from fMRI

Visualizing the perceptual content by analyzing human functional magneti...

CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training

We present variational generative adversarial networks, a general learni...

Unsupervised Deep Representations for Learning Audience Facial Behaviors

In this paper, we present an unsupervised learning approach for analyzin...

Replicating Active Appearance Model by Generator Network

A recent Cell paper [Chang and Tsao, 2017] reports an interesting discov...

Please sign up or login with your details

Forgot password? Click here to reset