What You See is What You Grasp: User-Friendly Grasping Guided by Near-eye-tracking

by   Shaochen Wang, et al.

This work presents a next-generation human-robot interface that can infer and realize the user's manipulation intention via sight only. Specifically, we develop a system that integrates near-eye-tracking and robotic manipulation to enable user-specified actions (e.g., grasp, pick-and-place, etc), where visual information is merged with human attention to create a mapping for desired robot actions. To enable sight guided manipulation, a head-mounted near-eye-tracking device is developed to track the eyeball movements in real-time, so that the user's visual attention can be identified. To improve the grasping performance, a transformer based grasp model is then developed. Stacked transformer blocks are used to extract hierarchical features where the volumes of channels are expanded at each stage while squeezing the resolution of feature maps. Experimental validation demonstrates that the eye-tracking system yields low gaze estimation error and the grasping system yields promising results on multiple grasping datasets. This work is a proof of concept for gaze interaction-based assistive robot, which holds great promise to help the elder or upper limb disabilities in their daily lives. A demo video is available at <https://www.youtube.com/watch?v=yuZ1hukYUrM>.


page 1

page 2

page 5

page 6


Addressing the eye-fixation problem in gaze tracking for human computer interface using the Vestibulo-ocular Reflex

A custom head-mounted system to track smooth eye movements for control o...

When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection

In this paper, we present a transformer-based architecture, namely TF-Gr...

MoGaze: A Dataset of Full-Body Motions that Includes Workspace Geometry and Eye-Gaze

As robots become more present in open human environments, it will become...

i-GSI: A Fast and Reliable Grasp-type Switching Interface based on Augmented Reality and Eye-tracking

The control of multi-fingered dexterous prosthetics hand remains challen...

Towards Intention Prediction for Handheld Robots: a Case of Simulated Block Copying

Within this work, we explore intention inference for user actions in the...

6-DoF Robotic Grasping with Transformer

Robotic grasping aims to detect graspable points and their corresponding...

Non-invasive Cognitive-level Human Interfacing for the Robotic Restoration of Reaching Grasping

Assistive and Wearable Robotics have the potential to support humans wit...

Please sign up or login with your details

Forgot password? Click here to reset