Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs

by   Furkan Ozcelik, et al.

Reconstructing perceived natural images from fMRI signals is one of the most engaging topics of neural decoding research. Prior studies had success in reconstructing either the low-level image features or the semantic/high-level aspects, but rarely both. In this study, we utilized an Instance-Conditioned GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate semantic attributes and preserved low-level details. The IC-GAN model takes as input a 119-dim noise vector and a 2048-dim instance feature vector extracted from a target image via a self-supervised learning model (SwAV ResNet-50); these instance features act as a conditioning for IC-GAN image generation, while the noise vector introduces variability between samples. We trained ridge regression models to predict instance features, noise vectors, and dense vectors (the output of the first dense layer of the IC-GAN generator) of stimuli from corresponding fMRI patterns. Then, we used the IC-GAN generator to reconstruct novel test images based on these fMRI-predicted variables. The generated images presented state-of-the-art results in terms of capturing the semantic attributes of the original test images while remaining relatively faithful to low-level image details. Finally, we use the learned regression model and the IC-GAN generator to systematically explore and visualize the semantic features that maximally drive each of several regions-of-interest in the human brain.


page 1

page 5

page 6

page 7


BigGAN-based Bayesian reconstruction of natural images from human brain activity

In the visual decoding domain, visually reconstructing presented images ...

Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion

In neural decoding research, one of the most intriguing topics is the re...

Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

Decoding and reconstructing images from brain imaging data is a research...

Decoding natural image stimuli from fMRI data with a surface-based convolutional network

Due to the low signal-to-noise ratio and limited resolution of functiona...

Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning

Reconstructing perceived images based on brain signals measured with fun...

Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Reconstructing seeing images from fMRI recordings is an absorbing resear...

Categorical Difference and Related Brain Regions of the Attentional Blink Effect

Attentional blink (AB) is a biological effect, showing that for 200 to 5...

Please sign up or login with your details

Forgot password? Click here to reset