A RelEntLess Benchmark for Modelling Graded Relations between Named Entities

by   Asahi Ushio, et al.

Relations such as "is influenced by", "is known for" or "is a competitor of" are inherently graded: we can rank entity pairs based on how well they satisfy these relations, but it is hard to draw a line between those pairs that satisfy them and those that do not. Such graded relations play a central role in many applications, yet they are typically not covered by existing Knowledge Graphs. In this paper, we consider the possibility of using Large Language Models (LLMs) to fill this gap. To this end, we introduce a new benchmark, in which entity pairs have to be ranked according to how much they satisfy a given graded relation. The task is formulated as a few-shot ranking problem, where models only have access to a description of the relation and five prototypical instances. We use the proposed benchmark to evaluate state-of-the-art relation embedding strategies as well as several recent LLMs, covering both publicly available LLMs and closed models such as GPT-4. Overall, we find a strong correlation between model size and performance, with smaller Language Models struggling to outperform a naive baseline. The results of the largest Flan-T5 and OPT models are remarkably strong, although a clear gap with human performance remains.


page 1

page 2

page 3

page 4


Latent Relation Language Models

In this paper, we propose Latent Relation Language Models (LRLMs), a cla...

Few-Shot Knowledge Graph Completion

Knowledge graphs (KGs) serve as useful resources for various natural lan...

Entity-Assisted Language Models for Identifying Check-worthy Sentences

We propose a new uniform framework for text classification and ranking t...

Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs

Knowledge Graphs (KGs) play a crucial role in enhancing e-commerce syste...

Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models

Pretrained language models (PLMs) for data-to-text (D2T) generation can ...

Exploiting Transitivity Constraints for Entity Matching in Knowledge Graphs

The goal of entity matching in knowledge graphs is to identify entities ...

PCRED: Zero-shot Relation Triplet Extraction with Potential Candidate Relation Selection and Entity Boundary Detection

Zero-shot relation triplet extraction (ZeroRTE) aims to extract relation...

Please sign up or login with your details

Forgot password? Click here to reset