Multi-modal Ensemble Classification for Generalized Zero Shot Learning

by   Rafael Felix, et al.

Generalized zero shot learning (GZSL) is defined by a training process containing a set of visual samples from seen classes and a set of semantic samples from seen and unseen classes, while the testing process consists of the classification of visual samples from seen and unseen classes. Current approaches are based on testing processes that focus on only one of the modalities (visual or semantic), even when the training uses both modalities (mostly for regularizing the training process). This under-utilization of modalities, particularly during testing, can hinder the classification accuracy of the method. In addition, we note a scarce attention to the development of learning methods that explicitly optimize a balanced performance of seen and unseen classes. Such issue is one of the reasons behind the vastly superior classification accuracy of seen classes in GZSL methods. In this paper, we mitigate these issues by proposing a new GZSL method based on multi-modal training and testing processes, where the optimization explicitly promotes a balanced classification accuracy between seen and unseen classes. Furthermore, we explore Bayesian inference for the visual and semantic classifiers, which is another novelty of our work in the GZSL framework. Experiments show that our method holds the state of the art (SOTA) results in terms of harmonic mean (H-mean) classification between seen and unseen classes and area under the seen and unseen curve (AUSUC) on several public GZSL benchmarks.


page 1

page 2

page 3

page 4


Multi-modal Cycle-consistent Generalized Zero-Shot Learning

In generalized zero shot learning (GZSL), the set of classes are split i...

A Semantics-Guided Class Imbalance Learning Model for Zero-Shot Classification

Zero-Shot Classification (ZSC) equips the learned model with the ability...

Generalised Zero-Shot Learning with Domain Classification in a Joint Semantic and Visual Space

Generalised zero-shot learning (GZSL) is a classification problem where ...

Improving Generalized Zero-Shot Learning by Semantic Discriminator

It is a recognized fact that the classification accuracy of unseen class...

Generalised Zero-Shot Learning with a Classifier Ensemble over Multi-Modal Embedding Spaces

Generalised zero-shot learning (GZSL) methods aim to classify previously...

Informed Democracy: Voting-based Novelty Detection for Action Recognition

Novelty detection is crucial for real-life applications. While it is com...

Continuous representations of intents for dialogue systems

Intent modelling has become an important part of modern dialogue systems...

Please sign up or login with your details

Forgot password? Click here to reset