Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading

by   Shuwen Deng, et al.

Eye movements during reading offer insights into both the reader's cognitive processes and the characteristics of the text that is being read. Hence, the analysis of scanpaths in reading have attracted increasing attention across fields, ranging from cognitive science over linguistics to computer science. In particular, eye-tracking-while-reading data has been argued to bear the potential to make machine-learning-based language models exhibit a more human-like linguistic behavior. However, one of the main challenges in modeling human scanpaths in reading is their dual-sequence nature: the words are ordered following the grammatical rules of the language, whereas the fixations are chronologically ordered. As humans do not strictly read from left-to-right, but rather skip or refixate words and regress to previous words, the alignment of the linguistic and the temporal sequence is non-trivial. In this paper, we develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. The alignment of the two sequences is achieved by a cross-sequence attention mechanism. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths. We provide an extensive within- and across-data set evaluation on different languages. An ablation study and qualitative analysis support an in-depth understanding of the model's behavior.


page 5

page 14


Modeling Human Reading with Neural Attention

When humans read text, they fixate some words and skip others. However, ...

Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models

Eye movements are known to reflect cognitive processes in reading, and p...

Integrating large language models and active inference to understand eye movements in reading and dyslexia

We present a novel computational model employing hierarchical active inf...

Modeling Task Effects in Human Reading with Neural Attention

Humans read by making a sequence of fixations and saccades. They often s...

Topic Modeling the Reading and Writing Behavior of Information Foragers

The general problem of "information foraging" in an environment about wh...

Enhancing Keyphrase Extraction from Microblogs using Human Reading Time

The premise of manual keyphrase annotation is to read the corresponding ...

Smartajweed Automatic Recognition of Arabic Quranic Recitation Rules

Tajweed is a set of rules to read the Quran in a correct Pronunciation o...

Please sign up or login with your details

Forgot password? Click here to reset