Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

by   Yunlu Yan, et al.

While multi-modal learning has been widely used for MRI reconstruction, it relies on paired multi-modal data which is difficult to acquire in real clinical scenarios. Especially in the federated setting, the common situation is that several medical institutions only have single-modal data, termed the modality missing issue. Therefore, it is infeasible to deploy a standard federated learning framework in such conditions. In this paper, we propose a novel communication-efficient federated learning framework, namely Fed-PMG, to address the missing modality challenge in federated multi-modal MRI reconstruction. Specifically, we utilize a pseudo modality generation mechanism to recover the missing modality for each single-modal client by sharing the distribution information of the amplitude spectrum in frequency space. However, the step of sharing the original amplitude spectrum leads to heavy communication costs. To reduce the communication cost, we introduce a clustering scheme to project the set of amplitude spectrum into finite cluster centroids, and share them among the clients. With such an elaborate design, our approach can effectively complete the missing modality within an acceptable communication cost. Extensive experiments demonstrate that our proposed method can attain similar performance with the ideal scenario, i.e., all clients have the full set of modalities. The source code will be released.


page 1

page 6

page 8


Cross-Modal Vertical Federated Learning for MRI Reconstruction

Federated learning enables multiple hospitals to cooperatively learn a s...

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

Recently, foundation models have exhibited remarkable advancements in mu...

FedMed-GAN: Federated Multi-Modal Unsupervised Brain Image Synthesis

Utilizing the paired multi-modal neuroimaging data has been proved to be...

Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning

The development of federated learning (FL) methods, which aim to learn f...

A Learnable Variational Model for Joint Multimodal MRI Reconstruction and Synthesis

Generating multi-contrasts/modal MRI of the same anatomy enriches diagno...

PIMMS: Permutation Invariant Multi-Modal Segmentation

In a research context, image acquisition will often involve a pre-define...

Please sign up or login with your details

Forgot password? Click here to reset