VideoDex: Learning Dexterity from Internet Videos

by   Kenneth Shaw, et al.

To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior. These actions and physical priors in the neural network dictate the typical human behavior for a particular robot task. We test our approach on a robot arm and dexterous hand-based system and show strong results on various manipulation tasks, outperforming various state-of-the-art methods. Videos at


page 2

page 3

page 4

page 5

page 6

page 16


Structured World Models from Human Videos

We tackle the problem of learning complex, general behaviors directly in...

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

We build a system that enables any human to control a robot hand and arm...

What You Say Is What You Show: Visual Narration Detection in Instructional Videos

Narrated "how-to" videos have emerged as a promising data source for a w...

Visual Prediction of Priors for Articulated Object Interaction

Exploration in novel settings can be challenging without prior experienc...

Affordances from Human Videos as a Versatile Representation for Robotics

Building a robot that can understand and learn to interact by watching h...

Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations

Videos that are shot using commodity hardware such as phones and surveil...

Egocentric scene context for human-centric environment understanding from video

First-person video highlights a camera-wearer's activities in the contex...

Please sign up or login with your details

Forgot password? Click here to reset