MEDIMP: Medical Images and Prompts for renal transplant representation learning

by   Leo Milecki, et al.

Renal transplantation emerges as the most effective solution for end-stage renal disease. Occurring from complex causes, a substantial risk of transplant chronic dysfunction persists and may lead to graft loss. Medical imaging plays a substantial role in renal transplant monitoring in clinical practice. However, graft supervision is multi-disciplinary, notably joining nephrology, urology, and radiology, while identifying robust biomarkers from such high-dimensional and complex data for prognosis is challenging. In this work, taking inspiration from the recent success of Large Language Models (LLMs), we propose MEDIMP – Medical Images and Prompts – a model to learn meaningful multi-modal representations of renal transplant Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE MRI) by incorporating structural clinicobiological data after translating them into text prompts. MEDIMP is based on contrastive learning from joint text-image paired embeddings to perform this challenging task. Moreover, we propose a framework that generates medical prompts using automatic textual data augmentations from LLMs. Our goal is to learn meaningful manifolds of renal transplant DCE MRI, interesting for the prognosis of the transplant or patient status (2, 3, and 4 years after the transplant), fully exploiting the available multi-modal data in the most efficient way. Extensive experiments and comparisons with other renal transplant representation learning methods with limited data prove the effectiveness of MEDIMP in a relevant clinical setting, giving new directions toward medical prompts. Our code is available at


page 1

page 2

page 3

page 4


Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation

Vision-Language Pretraining (VLP) has demonstrated remarkable capabiliti...

Heterogeneous Graph Learning for Multi-modal Medical Data Analysis

Routine clinical visits of a patient produce not only image data, but al...

Video-Text Representation Learning via Differentiable Weak Temporal Alignment

Learning generic joint representations for video and text by a supervise...

MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data

Mild cognitive impairment (MCI) conversion prediction, i.e., identifying...

Synthesis-based Imaging-Differentiation Representation Learning for Multi-Sequence 3D/4D MRI

Multi-sequence MRIs can be necessary for reliable diagnosis in clinical ...

Multi-Modal MRI Reconstruction with Spatial Alignment Network

In clinical practice, magnetic resonance imaging (MRI) with multiple con...

Spectral Decomposition in Deep Networks for Segmentation of Dynamic Medical Images

Dynamic contrast-enhanced magnetic resonance imaging (DCE- MRI) is a wid...

Please sign up or login with your details

Forgot password? Click here to reset