Cross-Modal Vertical Federated Learning for MRI Reconstruction

by   Yunlu Yan, et al.

Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure. Existing methods often take a common assumption that the data from different hospitals have the same modalities. However, such a setting is difficult to fully satisfy in practical applications, since the imaging guidelines may be different between hospitals, which makes the number of individuals with the same set of modalities limited. To this end, we formulate this practical-yet-challenging cross-modal vertical federated learning task, in which shape data from multiple hospitals have different modalities with a small amount of multi-modality data collected from the same individuals. To tackle such a situation, we develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction by effectively exploring the overlapping samples (individuals with multi-modalities) and solving the domain shift problem caused by different modalities. Particularly, our Fed-CRFD involves an intra-client feature disentangle scheme to decouple data into modality-invariant and modality-specific features, where the modality-invariant features are leveraged to mitigate the domain shift problem. In addition, a cross-client latent representation consistency constraint is proposed specifically for the overlapping samples to further align the modality-invariant features extracted from different modalities. Hence, our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem. Extensive experiments on two typical MRI datasets demonstrate that our network clearly outperforms state-of-the-art MRI reconstruction methods. The source code will be publicly released upon the publication of this work.


page 1

page 2

page 5

page 8

page 9

page 10


Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

While multi-modal learning has been widely used for MRI reconstruction, ...

Cross-modal Center Loss

Cross-modal retrieval aims to learn discriminative and modal-invariant f...

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation

Multimodal learning has seen great success mining data features from mul...

Prototype-Based Layered Federated Cross-Modal Hashing

Recently, deep cross-modal hashing has gained increasing attention. Howe...

Federated Multi-organ Segmentation with Partially Labeled Data

Federated learning is an emerging paradigm allowing large-scale decentra...

Unified cross-modality feature disentangler for unsupervised multi-domain MRI abdomen organs segmentation

Our contribution is a unified cross-modality feature disentagling approa...

Brain Image Synthesis with Unsupervised Multivariate Canonical CSCℓ_4Net

Recent advances in neuroscience have highlighted the effectiveness of mu...

Please sign up or login with your details

Forgot password? Click here to reset