Visual-Semantic Contrastive Alignment for Few-Shot Image Classification

by   Mohamed Afham, et al.

Few-Shot learning aims to train and optimize a model that can adapt to unseen visual classes with only a few labeled examples. The existing few-shot learning (FSL) methods, heavily rely only on visual data, thus fail to capture the semantic attributes to learn a more generalized version of the visual concept from very few examples. However, it is a known fact that human visual learning benefits immensely from inputs from multiple modalities such as vision, language, and audio. Inspired by the human learning nature of encapsulating the existing knowledge of a visual category which is in the form of language, we introduce a contrastive alignment mechanism for visual and semantic feature vectors to learn much more generalized visual concepts for few-shot learning. Our method simply adds an auxiliary contrastive learning objective which captures the contextual knowledge of a visual category from a strong textual encoder in addition to the existing training mechanism. Hence, the approach is more generalized and can be plugged into any existing FSL method. The pre-trained semantic feature extractor (learned from a large-scale text corpora) we use in our approach provides a strong contextual prior knowledge to assist FSL. The experimental results done in popular FSL datasets show that our approach is generic in nature and provides a strong boost to the existing FSL baselines.


page 1

page 2

page 3

page 4


Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning

We propose a novel framework for few-shot learning by leveraging large-s...

SEGA: Semantic Guided Attention on Visual Prototype for Few-Shot Learning

Teaching machines to recognize a new category based on few training samp...

Rich Semantics Improve Few-shot Learning

Human learning benefits from multi-modal inputs that often appear as ric...

Knowledge-augmented Few-shot Visual Relation Detection

Visual Relation Detection (VRD) aims to detect relationships between obj...

Learning Dynamic Alignment via Meta-filter for Few-shot Learning

Few-shot learning (FSL), which aims to recognise new classes by adapting...

Alignment Based Matching Networks for One-Shot Classification and Open-Set Recognition

Deep learning for object classification relies heavily on convolutional ...

Few-shot Single-view 3D Reconstruction with Memory Prior Contrastive Network

3D reconstruction of novel categories based on few-shot learning is appe...

Please sign up or login with your details

Forgot password? Click here to reset