research
∙
09/14/2023
Anchor Points: Benchmarking Models with Much Fewer Examples
Modern language models often exhibit powerful but brittle behavior, lead...
share
research
∙
05/24/2022
How Human is Human Evaluation? Improving the Gold Standard for NLG with Utility Theory
Human ratings are treated as the gold standard in NLG evaluation. The st...
research
∙
05/10/2022
Richer Countries and Richer Representations
We examine whether some countries are more richly represented in embeddi...
research
∙
05/10/2022
Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
Cosine similarity of contextual embeddings is used in many NLP tasks (e....
research
∙
09/19/2021
Conditional probing: measuring usable information beyond a baseline
Probing experiments investigate the extent to which neural representatio...
research
∙
05/31/2021
Attention Flows are Shapley Value Explanations
Shapley Values, a solution to the credit assignment problem in cooperati...
research
∙
04/17/2021
Frequency-based Distortions in Contextualized Word Embeddings
How does word frequency in pre-training data affect the behavior of simi...
research
∙
09/29/2020
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Benchmarks such as GLUE have helped drive advances in NLP by incentivizi...
research
∙
04/27/2020
BLEU Neighbors: A Reference-less Approach to Automatic Evaluation
Evaluation is a bottleneck in the development of natural language genera...
research
∙
04/26/2020
Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds
Most NLP datasets are not annotated with protected attributes such as ge...
research
∙
09/02/2019
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Replacing static word embeddings with contextualized word representation...
research
∙
09/02/2019
Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space
A notable property of word embeddings is that word relationships can exi...
research
∙
08/18/2019
Understanding Undesirable Word Embedding Associations
Word embeddings are often criticized for capturing undesirable word asso...
research
∙
10/11/2018