Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

by   Aravind Sivakumar, et al.

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots that learn to act autonomously in the real world. Videos at


page 1

page 2

page 3

page 4

page 6

page 8


VideoDex: Learning Dexterity from Internet Videos

To build general robotic agents that can operate in many environments, i...

A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU

In this paper, we present a multimodal mobile teleoperation system that ...

Structured World Models from Human Videos

We tackle the problem of learning complex, general behaviors directly in...

Learning from Demonstration with Weakly Supervised Disentanglement

Robotic manipulation tasks, such as wiping with a soft sponge, require c...

Quasi-Direct Drive for Low-Cost Compliant Robotic Manipulation

Robots must cost less and be force-controlled to enable widespread, safe...

Controlling by Showing: i-Mimic: A Video-based Method to Control Robotic Arms

A novel concept of vision-based intelligent control of robotic arms is d...

Put the Bear on the Chair! Intelligent Robot Interaction with Previously Unseen Objects via Robot Imagination

In this letter, we study the problem of autonomously placing a teddy bea...

Please sign up or login with your details

Forgot password? Click here to reset