Zero-Shot Text Classification via Self-Supervised Tuning

by   Chaoqun Liu, et al.

Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for meta-tuning. In this work, we propose a new paradigm based on self-supervised learning to solve zero-shot text classification tasks by tuning the language models with unlabeled data, called self-supervised tuning. By exploring the inherent structure of free texts, we propose a new learning objective called first sentence prediction to bridge the gap between unlabeled data and text classification tasks. After tuning the model to learn to predict the first sentence in a paragraph based on the rest, the model is able to conduct zero-shot inference on unseen tasks such as topic classification and sentiment analysis. Experimental results show that our model outperforms the state-of-the-art baselines on 7 out of 10 tasks. Moreover, the analysis reveals that our model is less sensitive to the prompt design. Our code and pre-trained models are publicly available at .


page 1

page 2

page 3

page 4


Zero-Shot Text Classification with Self-Training

Recent advances in large pretrained language models have increased atten...

Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective

We propose a new paradigm for zero-shot learners that is format agnostic...

SelfMix: Robust Learning Against Textual Label Noise with Self-Mixup Training

The conventional success of textual classification relies on annotated d...

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

State-of-the-art brain-to-text systems have achieved great success in de...

Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization

Training language models to learn from human instructions for zero-shot ...

Perturbation-based Self-supervised Attention for Attention Bias in Text Classification

In text classification, the traditional attention mechanisms usually foc...

Exploring Lottery Prompts for Pre-trained Language Models

Consistently scaling pre-trained language models (PLMs) imposes substant...

Please sign up or login with your details

Forgot password? Click here to reset