Metadata Shaping: Natural Language Annotations for the Tail

by   Simran Arora, et al.

Language models (LMs) have made remarkable progress, but still struggle to generalize beyond the training data to rare linguistic patterns. Since rare entities and facts are prevalent in the queries users submit to popular applications such as search and personal assistant systems, improving the ability of LMs to reliably capture knowledge over rare entities is a pressing challenge studied in significant prior work. Noticing that existing approaches primarily modify the LM architecture or introduce auxiliary objectives to inject useful entity knowledge, we ask to what extent we could match the quality of these architectures using a base LM architecture, and only changing the data? We propose metadata shaping, a method in which readily available metadata, such as entity descriptions and categorical tags, are appended to examples based on information theoretic metrics. Intuitively, if metadata corresponding to popular entities overlap with metadata for rare entities, the LM may be able to better reason about the rare entities using patterns learned from similar popular entities. On standard entity-rich tasks (TACRED, FewRel, OpenEntity), with no changes to the LM whatsoever, metadata shaping exceeds the BERT-baseline by up to 5.3 F1 points, and achieves or competes with state-of-the-art results. We further show the improvements are up to 10x larger on examples containing tail versus popular entities.


page 1

page 2

page 3

page 4


Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation

A challenge for named entity disambiguation (NED), the task of mapping t...

Entity-aware Transformers for Entity Search

Pre-trained language models such as BERT have been a key ingredient to a...

FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge

Evaluating the factual consistency of automatically generated summaries ...

Personalised Language Modelling of Screen Characters Using Rich Metadata Annotations

Personalisation of language models for dialogue sensitises them to bette...

A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

Pre-trained language models (PLMs) cannot well recall rich factual knowl...

Knowledge Base Completion for Long-Tail Entities

Despite their impressive scale, knowledge bases (KBs), such as Wikidata,...

Learning Hidden Structures with Relational Models by Adequately Involving Rich Information in A Network

Effectively modelling hidden structures in a network is very practical b...

Please sign up or login with your details

Forgot password? Click here to reset