Language Models and Word Sense Disambiguation: An Overview and Analysis

by   Daniel Loureiro, et al.

Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations for encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT performs a decent job in capturing high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data.


page 18

page 24

page 38


Incorporating Word Sense Disambiguation in Neural Language Models

We present two supervised (pre-)training methods to incorporate gloss de...

What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories

Language Models are the core for almost any Natural Language Processing ...

SenseBERT: Driving Some Sense into BERT

Self-supervision techniques have allowed neural language models to advan...

A Comparative Study of Lexical Substitution Approaches based on Neural Language Models

Lexical substitution in context is an extremely powerful technology that...

GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

Word Sense Disambiguation (WSD) aims to find the exact sense of an ambig...

Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation

Word sense disambiguation (WSD), which aims to determine an appropriate ...

Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability

With the recent influx of bidirectional contextualized transformer langu...

Please sign up or login with your details

Forgot password? Click here to reset