Counterfactual Zero-Shot and Open-Set Visual Recognition

by   Zhongqi Yue, et al.

We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If “yes”, the sample is from a certain class, and “no” otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize. Codes are available at


page 8

page 16

page 17


A Generative Framework for Zero-Shot Learning with Adversarial Domain Adaptation

In this paper, we present a domain adaptation based generative framework...

Visual and Semantic Prototypes-Jointly Guided CNN for Generalized Zero-shot Learning

In the process of exploring the world, the curiosity constantly drives h...

Convolutional Prototype Learning for Zero-Shot Recognition

Zero-shot learning (ZSL) has received increasing attention in recent yea...

Contrastive Embedding for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) aims to recognize objects from bot...

How Robust are Discriminatively Trained Zero-Shot Learning Models?

Data shift robustness has been primarily investigated from a fully super...

Mitigating Generation Shifts for Generalized Zero-Shot Learning

Generalized Zero-Shot Learning (GZSL) is the task of leveraging semantic...

A Semantics-Guided Class Imbalance Learning Model for Zero-Shot Classification

Zero-Shot Classification (ZSC) equips the learned model with the ability...

Please sign up or login with your details

Forgot password? Click here to reset