MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning

by   Bohan Wu, et al.

Vision-based grasping systems typically adopt an open-loop execution of a planned grasp. This policy can fail due to many reasons, including ubiquitous calibration error. Recovery from a failed grasp is further complicated by visual occlusion, as the hand is usually occluding the vision sensor as it attempts another open-loop regrasp. This work presents MAT, a tactile closed-loop method capable of realizing grasps provided by a coarse initial positioning of the hand above an object. Our algorithm is a deep reinforcement learning (RL) policy optimized through the clipped surrogate objective within a maximum entropy RL framework to balance exploitation and exploration. The method utilizes tactile and proprioceptive information to act through both fine finger motions and larger regrasp movements to execute stable grasps. A novel curriculum of action motion magnitude makes learning more tractable and helps turn common failure cases into successes. Careful selection of features that exhibit small sim-to-real gaps enables this tactile grasping policy, trained purely in simulation, to transfer well to real world environments without the need for additional learning. Experimentally, this methodology improves over a vision-only grasp success rate substantially on a multi-fingered robot hand. When this methodology is used to realize grasps from coarse initial positions provided by a vision-only planner, the system is made dramatically more robust to calibration errors in the camera-robot transform.


page 2

page 4

page 6

page 8

page 15


QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

In this paper, we study the problem of learning vision-based dynamic man...

Towards Learning to Detect and Predict Contact Events on Vision-based Tactile Sensors

In essence, successful grasp boils down to correct responses to multiple...

End-to-end grasping policies for human-in-the-loop robots via deep reinforcement learning

State-of-the-art human-in-the-loop robot grasping is hugely suffered by ...

Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes

Recent advances in on-policy reinforcement learning (RL) methods enabled...

Tactile Grasp Refinement using Deep Reinforcement Learning and Analytic Grasp Stability Metrics

Reward functions are at the heart of every reinforcement learning (RL) a...

Haptic search with the Smart Suction Cup on adversarial objects

Suction cups are an important gripper type in industrial robot applicati...

Grasping Using Tactile Sensing and Deep Calibration

Tactile perception is an essential ability of intelligent robots in inte...

Please sign up or login with your details

Forgot password? Click here to reset