WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

by   Anna Breit, et al.
Iran University of Science and Technology
Cardiff University

In this paper, we present WiC-TSV (Target Sense Verification for Words in Context), a new multi-domain evaluation benchmark for Word Sense Disambiguation (WSD) and Entity Linking (EL). Our benchmark is different from conventional WSD and EL benchmarks for it being independent of a general sense inventory, making it highly flexible for the evaluation of a diverse set of models and systems in different domains. WiC-TSV is split into three tasks (systems get hypernymy or definitional or both hypernymy and definitional information about the target sense). Test data is available in four domains: general (WordNet), computer science, cocktails and medical concepts. Results show that existing state-of-the-art language models such as BERT can achieve a high performance in both in-domain data and out-of-domain data, but they still have room for improvement. WiC-TSV task data is available at <https://competitions.codalab.org/competitions/23683>.


page 1

page 2

page 3

page 4


Incorporating Word Sense Disambiguation in Neural Language Models

We present two supervised (pre-)training methods to incorporate gloss de...

XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

The ability to correctly model distinct meanings of a word is crucial fo...

Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution

Lexical substitution, i.e. generation of plausible words that can replac...

Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences

Domain adaptation or transfer learning using pre-trained language models...

Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation

In this article, we tackle the issue of the limited quantity of manually...

That's sick dude!: Automatic identification of word sense change across different timescales

In this paper, we propose an unsupervised method to identify noun sense ...

Scalable Performance Analysis for Vision-Language Models

Joint vision-language models have shown great performance over a diverse...

Please sign up or login with your details

Forgot password? Click here to reset