Towards General Purpose Medical AI: Continual Learning Medical Foundation Model

by   Huahui Yi, et al.

Inevitable domain and task discrepancies in real-world scenarios can impair the generalization performance of the pre-trained deep models for medical data. Therefore, we audaciously propose that we should build a general-purpose medical AI system that can be seamlessly adapted to downstream domains/tasks. Since the domain/task adaption procedures usually involve additional labeling work for the target data, designing a data-efficient adaption algorithm is desired to save the cost of transferring the learned knowledge. Our recent work found that vision-language models (VLMs) are efficient learners with extraordinary cross-domain ability. Therefore, in this work, we further explore the possibility of leveraging pre-trained VLMs as medical foundation models for building general-purpose medical AI, where we thoroughly investigate three machine-learning paradigms, i.e., domain/task-specialized learning, joint learning, and continual learning, for training the VLMs and evaluate their generalization performance on cross-domain and cross-task test sets. To alleviate the catastrophic forgetting during sequential training, we employ rehearsal learning and receive a sharp boost in terms of generalization capability. In a nutshell, our empirical evidence suggests that continual learning may be a practical and efficient learning paradigm for the medical foundation model. And we hope researchers can use our empirical evidence as basement to further explore the path toward medical foundation model.


page 6

page 9


Investigating Forgetting in Pre-Trained Representations Through Continual Learning

Representation forgetting refers to the drift of contextualized represen...

CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering

Visual Question Answering (VQA) is a multi-discipline research task. To ...

A Comprehensive Survey of Continual Learning: Theory, Method and Application

To cope with real-world dynamics, an intelligent agent needs to incremen...

Rehearsal-Free Domain Continual Face Anti-Spoofing: Generalize More and Forget Less

Face Anti-Spoofing (FAS) is recently studied under the continual learnin...

Overcoming General Knowledge Loss with Selective Parameter Finetuning

Foundation models encompass an extensive knowledge base and offer remark...

POP: Prompt Of Prompts for Continual Learning

Continual learning (CL) has attracted increasing attention in the recent...

A Continual Learning Approach for Cross-Domain White Blood Cell Classification

Accurate classification of white blood cells in peripheral blood is esse...

Please sign up or login with your details

Forgot password? Click here to reset