Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes

by   Liang Xie, et al.

This paper proposes a learning-based visual peg-in-hole that enables training with several shapes in simulation, and adapting to arbitrary unseen shapes in real world with minimal sim-to-real cost. The core idea is to decouple the generalization of the sensory-motor policy to the design of a fast-adaptable perception module and a simulated generic policy module. The framework consists of a segmentation network (SN), a virtual sensor network (VSN), and a controller network (CN). Concretely, the VSN is trained to measure the pose of the unseen shape from a segmented image. After that, given the shape-agnostic pose measurement, the CN is trained to achieve generic peg-in-hole. Finally, when applying to real unseen holes, we only have to fine-tune the SN required by the simulated VSN+CN. To further minimize the transfer cost, we propose to automatically collect and annotate the data for the SN after one-minute human teaching. Simulated and real-world results are presented under the configurations of eye-to/in-hand. An electric vehicle charging system with the proposed policy inside achieves a 10/10 success rate in 2-3s, using only hundreds of auto-labeled samples for the SN transfer.


page 1

page 3

page 6

page 7

page 8

page 11

page 12

page 14


Learning to Fill the Seam by Vision: Sub-millimeter Peg-in-hole on Unseen Shapes in Real World

In the peg insertion task, human pays attention to the seam between the ...

Sim-to-Real Transfer of Accurate Grasping with Eye-In-Hand Observations and Continuous Control

In the context of deep learning for robotics, we show effective method o...

How to Close Sim-Real Gap? Transfer with Segmentation!

One fundamental difficulty in robotic learning is the sim-real gap probl...

Provable Sim-to-real Transfer in Continuous Domain with Partial Observations

Sim-to-real transfer trains RL agents in the simulated environments and ...

Virtual-to-Real: Learning to Control in Visual Semantic Segmentation

Collecting training data from the physical world is usually time-consumi...

Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

We address the problem of discovering 3D parts for objects in unseen cat...

Learn to Differ: Sim2Real Small Defection Segmentation Network

Recent studies on deep-learning-based small defection segmentation appro...

Please sign up or login with your details

Forgot password? Click here to reset