Unified Demonstration Retriever for In-Context Learning

by   Xiaonan Li, et al.

In-context learning is a new learning paradigm where a language model conditions on a few input-output pairs (demonstrations) and a test input, and directly outputs the prediction. It has been shown highly dependent on the provided demonstrations and thus promotes the research of demonstration retrieval: given a test input, relevant examples are retrieved from the training set to serve as informative demonstrations for in-context learning. While previous works focus on training task-specific retrievers for several tasks separately, these methods are often hard to transfer and scale on various tasks, and separately trained retrievers incur a lot of parameter storage and deployment cost. In this paper, we propose Unified Demonstration Retriever (UDR), a single model to retrieve demonstrations for a wide range of tasks. To train UDR, we cast various tasks' training signals into a unified list-wise ranking formulation by language model's feedback. Then we propose a multi-task list-wise ranking training framework, with an iterative mining strategy to find high-quality candidates, which can help UDR fully incorporate various tasks' signals. Experiments on 30+ tasks across 13 task families and multiple data domains show that UDR significantly outperforms baselines. Further analyses show the effectiveness of each proposed component and UDR's strong ability in various scenarios including different LMs (1.3B - 175B), unseen datasets, varying demonstration quantities, etc.


page 1

page 2

page 3

page 4


Dr.ICL: Demonstration-Retrieved In-context Learning

In-context learning (ICL), teaching a large language model (LLM) to perf...

Exploring Demonstration Ensembling for In-context Learning

In-context learning (ICL) operates by showing language models (LMs) exam...

Finding Supporting Examples for In-Context Learning

In-context learning is a new learning paradigm where a language model ob...

Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts

Anaphora resolution is an important task for information extraction acro...

It Takes One to Tango but More Make Trouble? In-Context Training with Different Number of Demonstrations

Large language models (LLMs) are capable to perform complex reasoning by...

Meta-training with Demonstration Retrieval for Efficient Few-shot Learning

Large language models show impressive results on few-shot NLP tasks. How...

Adversarial Demonstration Attacks on Large Language Models

With the emergence of more powerful large language models (LLMs), such a...

Please sign up or login with your details

Forgot password? Click here to reset