Rich Semantics Improve Few-shot Learning

by   Mohamed Afham, et al.

Human learning benefits from multi-modal inputs that often appear as rich semantics (e.g., description of an object's attributes while learning about it). This enables us to learn generalizable concepts from very limited visual examples. However, current few-shot learning (FSL) methods use numerical class labels to denote object classes which do not provide rich semantic meanings about the learned concepts. In this work, we show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the FSL performance. Given a support set and queries, our main idea is to create a bottleneck visual feature (hybrid prototype) which is then used to generate language descriptions of the classes as an auxiliary task during training. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens that can encode intricate relationships between the two modalities. Forcing the prototypes to retain semantic information about class description acts as a regularizer on the visual features, improving their generalization to novel classes at inference. Furthermore, this strategy imposes a human prior on the learned representations, ensuring that the model is faithfully relating visual and semantic concepts, thereby improving model interpretability. Our experiments on four datasets and ablation studies show the benefit of effectively modeling rich semantics for FSL.


page 1

page 4

page 7

page 11

page 12


Visual-Semantic Contrastive Alignment for Few-Shot Image Classification

Few-Shot learning aims to train and optimize a model that can adapt to u...

Baby steps towards few-shot learning with multiple semantics

Learning from one or few visual examples is one of the key capabilities ...

CANZSL: Cycle-Consistent Adversarial Networks for Zero-Shot Learning from Natural Language

Existing methods using generative adversarial approaches for Zero-Shot L...

Information Symmetry Matters: A Modal-Alternating Propagation Network for Few-Shot Learning

Semantic information provides intra-class consistency and inter-class di...

Multi-Modal Prototypes for Open-Set Semantic Segmentation

In semantic segmentation, adapting a visual system to novel object categ...

Few-shot Learning with Contextual Cueing for Object Recognition in Complex Scenes

Few-shot Learning aims to recognize new concepts from a small number of ...

Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning

The increasing impact of black box models, and particularly of unsupervi...

Please sign up or login with your details

Forgot password? Click here to reset