Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

by   Patrick Huber, et al.

With a growing number of BERTology work analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and fine-tuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distribution within and between models.


page 5

page 12

page 13

page 15

page 16


Assessing Discourse Relations in Language Generationfrom Pre-trained Language Models

Recent advances in NLP have been attributed to the emergence of large-sc...

Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

Discourse processing suffers from data sparsity, especially for dialogue...

A Side-by-side Comparison of Transformers for English Implicit Discourse Relation Classification

Though discourse parsing can help multiple NLP fields, there has been no...

Pragmatic competence of pre-trained language models through the lens of discourse connectives

As pre-trained language models (LMs) continue to dominate NLP, it is inc...

Document Classification for COVID-19 Literature

The global pandemic has made it more important than ever to quickly and ...

The future is different: Large pre-trained language models fail in prediction tasks

Large pre-trained language models (LPLM) have shown spectacular success ...

What GPT Knows About Who is Who

Coreference resolution – which is a crucial task for understanding disco...

Please sign up or login with your details

Forgot password? Click here to reset