Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

by   Forrest Davis, et al.
cornell university

A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.


Emergence of a phonological bias in ChatGPT

Current large language models, such as OpenAI's ChatGPT, have captured t...

Language Models Can Learn Exceptions to Syntactic Rules

Artificial neural networks can generalize productively to novel contexts...

Do RNNs learn human-like abstract word order preferences?

RNN language models have achieved state-of-the-art results on various ta...

Large Language Models Are Partially Primed in Pronoun Interpretation

While a large body of literature suggests that large language models (LL...

Pretraining on Non-linguistic Structure as a Tool for Analyzing Learning Bias in Language Models

We propose a novel methodology for analyzing the encoding of grammatical...

Testing AI performance on less frequent aspects of language reveals insensitivity to underlying meaning

Advances in computational methods and big data availability have recentl...

Please sign up or login with your details

Forgot password? Click here to reset