Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

by   Marco D'Alessandro, et al.

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a pre-defined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.


page 1

page 2

page 3

page 4


Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

Continual learning (CL) can help pre-trained vision-language models effi...

Learnability and Algorithm for Continual Learning

This paper studies the challenging continual learning (CL) setting of Cl...

PromptFusion: Decoupling Stability and Plasticity for Continual Learning

Continual learning refers to the capability of continuously learning fro...

Evolving Dictionary Representation for Few-shot Class-incremental Learning

New objects are continuously emerging in the dynamically changing world ...

Learnable Distribution Calibration for Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) faces challenges of memorizi...

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Continual learning requires incremental compatibility with a sequence of...

Image-Object-Specific Prompt Learning for Few-Shot Class-Incremental Learning

While many FSCIL studies have been undertaken, achieving satisfactory pe...

Please sign up or login with your details

Forgot password? Click here to reset