Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

by   Fan Yin, et al.

Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks. However, it remains unclear whether models truly understand task definitions and whether the human-written definitions are optimal. In this paper, we systematically study the role of task definitions in instruction learning. We first conduct an ablation analysis informed by human annotations to understand which parts of a task definition are most important, and find that model performance only drops substantially when removing contents describing the task output, in particular label information. Next, we propose an automatic algorithm to compress task definitions to a minimal supporting set of tokens, and find that 60% of tokens can be removed while maintaining or even improving model performance. Based on these results, we propose two strategies to help models better leverage task instructions: (1) providing only key information for tasks in a common structured format, and (2) adding a meta-tuning stage to help the model better understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L improvement over 119 unseen test tasks.


page 1

page 2

page 3

page 4


Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning

Recent works on instruction tuning (IT) have achieved great performance ...

Improving Cross-Task Generalization with Step-by-Step Instructions

Instruction tuning has been shown to be able to improve cross-task gener...

LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity

Cross-task generalization is a significant outcome that defines mastery ...

Exploring Format Consistency for Instruction Tuning

Instruction tuning has emerged as a promising approach to enhancing larg...

Robust Instruction Optimization for Large Language Models with Distribution Shifts

Large Language Models have demonstrated significant ability in accomplis...

PIVOINE: Instruction Tuning for Open-world Information Extraction

We consider the problem of Open-world Information Extraction (Open-world...

Please sign up or login with your details

Forgot password? Click here to reset