Sharing deep generative representation for perceived image reconstruction from human brain activity

by   Changde Du, et al.

Decoding human brain activities via functional magnetic resonance imaging (fMRI) has gained increasing attention in recent years. While encouraging results have been reported in brain states classification tasks, reconstructing the details of human visual experience still remains difficult. Two main challenges that hinder the development of effective models are the perplexing fMRI measurement noise and the high dimensionality of limited data instances. Existing methods generally suffer from one or both of these issues and yield dissatisfactory results. In this paper, we tackle this problem by casting the reconstruction of visual stimulus as the Bayesian inference of missing view in a multiview latent variable model. Sharing a common latent representation, our joint generative model of external stimulus and brain response is not only "deep" in extracting nonlinear features from visual images, but also powerful in capturing correlations among voxel activities of fMRI recordings. The nonlinearity and deep structure endow our model with strong representation ability, while the correlations of voxel activities are critical for suppressing noise and improving prediction. We devise an efficient variational Bayesian method to infer the latent variables and the model parameters. To further improve the reconstruction accuracy, the latent representations of testing instances are enforced to be close to that of their neighbours from the training set via posterior regularization. Experiments on three fMRI recording datasets demonstrate that our approach can more accurately reconstruct visual stimuli.


page 7

page 8


UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity

Image reconstruction and captioning from brain activity evoked by visual...

Mind Reader: Reconstructing complex images from brain activities

Understanding how the brain encodes external stimuli and how these stimu...

Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities

Decoding visual stimuli from neural responses recorded by functional Mag...

Natural Image Reconstruction from fMRI using Deep Learning: A Survey

With the advent of brain imaging techniques and machine learning tools, ...

Locality and low-dimensions in the prediction of natural experience from fMRI

Functional Magnetic Resonance Imaging (fMRI) provides dynamical access i...

Deep adversarial neural decoding

Here, we present a novel approach to solve the problem of reconstructing...

A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity

Reconstructing natural videos from fMRI brain recordings is very challen...

Please sign up or login with your details

Forgot password? Click here to reset