Towards End-to-end Video-based Eye-Tracking

by   Seonwook Park, et al.

Estimating eye-gaze from images alone is a challenging task, in large parts due to un-observable person-specific factors. Achieving high accuracy typically requires labeled data from test users which may not be attainable in real applications. We observe that there exists a strong relationship between what users are looking at and the appearance of the user's eyes. In response to this understanding, we propose a novel dataset and accompanying method which aims to explicitly learn these semantic and temporal relationships. Our video dataset consists of time-synchronized screen recordings, user-facing camera views, and eye gaze data, which allows for new benchmarks in temporal gaze tracking as well as label-free refinement of gaze. Importantly, we demonstrate that the fusion of information from visual stimuli as well as eye images can lead towards achieving performance similar to literature-reported figures acquired through supervised personalization. Our final method yields significant performance improvements on our proposed EVE dataset, with up to a 28 percent improvement in Point-of-Gaze estimates (resulting in 2.49 degrees in angular error), paving the path towards high-accuracy screen-based eye tracking purely from webcam sensors. The dataset and reference source code are available at


page 5

page 13

page 19

page 21


RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking

Accurate eye segmentation can improve eye-gaze estimation and support in...

Decoding Attention from Gaze: A Benchmark Dataset and End-to-End Models

Eye-tracking has potential to provide rich behavioral data about human c...

Event Based, Near Eye Gaze Tracking Beyond 10,000Hz

Fast and accurate eye tracking is crucial for many applications. Current...

Reinforcement learning for the manipulation of eye tracking data

In this paper, we present an approach based on reinforcement learning fo...

A New Robust Multivariate Mode Estimator for Eye-tracking Calibration

We propose in this work a new method for estimating the main mode of mul...

Watch to Edit: Video Retargeting using Gaze

We present a novel approach to optimally retarget videos for varied disp...

Please sign up or login with your details

Forgot password? Click here to reset