EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision

by   Siddharth Biswal, et al.

Deep generative models have enabled the automated synthesis of high-quality data for diverse applications. However, the most effective generative models are specialized to data from a single domain (e.g., images or text). Real-world applications such as healthcare require multi-modal data from multiple domains (e.g., both images and corresponding text), which are difficult to acquire due to limited availability and privacy concerns and are much harder to synthesize. To tackle this joint synthesis challenge, we propose an End-to-end MultImodal X-ray genERative model (EMIXER) for jointly synthesizing x-ray images and corresponding free-text reports, all conditional on diagnosis labels. EMIXER is an conditional generative adversarial model by 1) generating an image based on a label, 2) encoding the image to a hidden embedding, 3) producing the corresponding text via a hierarchical decoder from the image embedding, and 4) a joint discriminator for assessing both the image and the corresponding text. EMIXER also enables self-supervision to leverage vast amount of unlabeled data. Extensive experiments with real X-ray reports data illustrate how data augmentation using synthesized multimodal samples can improve the performance of a variety of supervised tasks including COVID-19 X-ray classification with very limited samples. The quality of generated images and reports are also confirmed by radiologists. We quantitatively show that EMIXER generated synthetic datasets can augment X-ray image classification, report generation models to achieve 5.94 data samples. Taken together, our results highlight the promise of state of generative models to advance clinical machine learning.


page 18

page 19

page 20


Automated Knee X-ray Report Generation

Gathering manually annotated images for the purpose of training a predic...

Adversarial Pulmonary Pathology Translation for Pairwise Chest X-ray Data Augmentation

Recent works show that Generative Adversarial Networks (GANs) can be suc...

Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment

Self-supervised learning provides an opportunity to explore unlabeled ch...

Implicit Data Augmentation Using Feature Interpolation for Diversified Low-Shot Image Generation

Training of generative models especially Generative Adversarial Networks...

Synthesizing Cough Audio with GAN for COVID-19 Detection

For this final year project, the goal is to add to the published works w...

Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models

We propose Retrieval Augmented Generation (RAG) as an approach for autom...

End-to-End Time-Lapse Video Synthesis from a Single Outdoor Image

Time-lapse videos usually contain visually appealing content but are oft...

Please sign up or login with your details

Forgot password? Click here to reset