Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rules

by   Priyanka Sukumaran, et al.
University of Bristol

LSTMs trained on next-word prediction can accurately perform linguistic tasks that require tracking long-distance syntactic dependencies. Notably, model accuracy approaches human performance on number agreement tasks (Gulordava et al., 2018). However, we do not have a mechanistic understanding of how LSTMs perform such linguistic tasks. Do LSTMs learn abstract grammatical rules, or do they rely on simple heuristics? Here, we test gender agreement in French which requires tracking both hierarchical syntactic structures and the inherent gender of lexical units. Our model is able to reliably predict long-distance gender agreement in two subject-predicate contexts: noun-adjective and noun-passive-verb agreement. The model showed more inaccuracies on plural noun phrases with gender attractors compared to singular cases, suggesting a reliance on clues from gendered articles for agreement. Overall, our study highlights key ways in which LSTMs deviate from human behaviour and questions whether LSTMs genuinely learn abstract syntactic rules and categories. We propose using gender agreement as a useful probe to investigate the underlying mechanisms, internal representations, and linguistic capabilities of LSTM language models.


The emergence of number and syntax units in LSTM language models

Recent work has shown that LSTMs trained on a generic language modeling ...

Numerical Sequence Prediction using Bayesian Concept Learning

When people learn mathematical patterns or sequences, they are able to i...

Attribution Analysis of Grammatical Dependencies in LSTMs

LSTM language models have been shown to capture syntax-sensitive grammat...

CLiMP: A Benchmark for Chinese Language Model Evaluation

Linguistically informed analyses of language models (LMs) contribute to ...

Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement

Many recent works have demonstrated that unsupervised sentence represent...

Exploring Processing of Nested Dependencies in Neural-Network Language Models and Humans

Recursive processing in sentence comprehension is considered a hallmark ...

Please sign up or login with your details

Forgot password? Click here to reset