research
∙
09/13/2023
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Most interpretability research in NLP focuses on understanding the behav...
share
research
∙
08/18/2023
Latent State Models of Training Dynamics
The impact of randomness on model training is poorly understood. How do ...
research
∙
05/24/2023
Dynamic Masking Rate Schedules for MLM Pretraining
Most works on transformers trained with the Masked Language Modeling (ML...
research
∙
11/17/2022
One Venue, Two Conferences: The Separation of Chinese and American Citation Networks
At NeurIPS, American and Chinese institutions cite papers from each othe...
research
∙
08/17/2022
Learning Transductions to Test Systematic Compositionality
Recombining known primitive concepts into larger novel combinations is a...
research
∙
05/24/2022
Linear Connectivity Reveals Generalization Strategies
It is widely accepted in the mode connectivity literature that when two ...
research
∙
06/30/2021
The MultiBERTs: BERT Reproductions for Robustness Analysis
Experiments with pretrained models such as BERT are often based on a sin...
research
∙
05/21/2021
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge – e.g...
research
∙
10/06/2020
LSTMs Compose (and Learn) Bottom-Up
Recent work in NLP shows that LSTM language models capture hierarchical ...
research
∙
10/05/2020
Pareto Probing: Trading Off Accuracy for Complexity
The question of how to probe contextual word representations in a way th...
research
∙
04/27/2020
Word Interdependence Exposes How LSTMs Compose Representations
Recent work in NLP shows that LSTM language models capture compositional...
research
∙
11/12/2019
How to Evaluate Word Representations of Informal Domain?
Diverse word representations have surged in most state-of-the-art natura...
research
∙
07/22/2019
Sparsity Emerges Naturally in Neural Language Models
Concerns about interpretability, computational resources, and principled...
research
∙
11/01/2018
Understanding Learning Dynamics Of Language Models with SVCCA
Recent work has demonstrated that neural language models encode linguist...
research
∙
01/15/2017
DyNet: The Dynamic Neural Network Toolkit
We describe DyNet, a toolkit for implementing neural network models base...
research
∙
06/27/2016