Learning to Reject with a Fixed Predictor: Application to Decontextualization

01/22/2023

∙

by   Christopher Mohri, et al.

∙

0

∙

share

We study the problem of classification with a reject option for a fixed predictor, applicable in natural language processing. where many correct labels are often possible We introduce a new problem formulation for this scenario, and an algorithm minimizing a new surrogate loss function. We provide a complete theoretical analysis of the surrogate loss function with a strong H-consistency guarantee. For evaluation, we choose the decontextualization task, and provide a manually-labelled dataset of 2,000 examples. Our algorithm significantly outperforms the baselines considered, with a ∼25% improvement in coverage when halving the error rate, which is only ∼ 3 % away from the theoretical limit.