A Dual-Arm Collaborative Framework for Dexterous Manipulation in Unstructured Environments with Contrastive Planning

by   Shengzeng Huo, et al.

Most object manipulation strategies for robots are based on the assumption that the object is rigid (i.e., with fixed geometry) and the goal's details have been fully specified (e.g., the exact target pose). However, there are many tasks that involve spatial relations in human environments where these conditions may be hard to satisfy, e.g., bending and placing a cable inside an unknown container. To develop advanced robotic manipulation capabilities in unstructured environments that avoid these assumptions, we propose a novel long-horizon framework that exploits contrastive planning in finding promising collaborative actions. Using simulation data collected by random actions, we learn an embedding model in a contrastive manner that encodes the spatio-temporal information from successful experiences, which facilitates the subgoal planning through clustering in the latent space. Based on the keypoint correspondence-based action parameterization, we design a leader-follower control scheme for the collaboration between dual arms. All models of our policy are automatically trained in simulation and can be directly transferred to real-world environments. To validate the proposed framework, we conduct a detailed experimental study on a complex scenario subject to environmental and reachability constraints in both simulation and real environments.


page 1

page 4

page 8

page 9

page 10


Dual-arm Coordinated Manipulation for Object Twisting with Human Intelligence

Robotic dual-arm twisting is a common but very challenging task in both ...

Collaborative Robotic Manipulation: A Use Case of Articulated Objects in Three-dimensions with Gravity

This paper addresses two intertwined needs for collaborative robots oper...

Keypoint-Based Bimanual Shaping of Deformable Linear Objects under Environmental Constraints using Hierarchical Action Planning

This paper addresses the problem of contact-based manipulation of deform...

DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Object Manipulation

It is essential yet challenging for future home-assistant robots to unde...

Autonomous Dual-Arm Manipulation of Familiar Objects

Autonomous dual-arm manipulation is an essential skill to deploy robots ...

Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning

Model-based reinforcement learning (MBRL) with real-time planning has sh...

Synthesizing Manipulation Sequences for Under-Specified Tasks using Unrolled Markov Random Fields

Many tasks in human environments require performing a sequence of naviga...

Please sign up or login with your details

Forgot password? Click here to reset