Multi-Fingered In-Hand Manipulation with Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

by   Satoshi Funabashi, et al.

Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize it for achieving manipulation skills is still an open issue. This paper presents a control method based on a graph convolutional network (GCN) which extracts geodesical features from the tactile data with complicated sensor alignments. Moreover, object property labels are provided to the GCN to adjust in-hand manipulation motions. Distributed tri-axial tactile sensors are mounted on the fingertips, finger phalanges and palm of an Allegro hand, resulting in 1152 tactile measurements. Training data is collected with a data-glove to transfer human dexterous manipulation directly to the robot hand. The GCN achieved high success rates for in-hand manipulation. We also confirmed that fragile objects were deformed less when correct object labels were provided to the GCN. When visualizing the activation of the GCN with a PCA, we verified that the network acquired geodesical features. Our method achieved stable manipulation even when an experimenter pulled a grasped object and for untrained objects.


page 1

page 5

page 6

page 7


TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot

In this paper, we propose a novel framework for tactile-based dexterous ...

Tactile-ViewGCN: Learning Shape Descriptor from Tactile Data using Graph Convolutional Network

For humans, our "senses of touch" have always been necessary for our abi...

Multi-Modal Geometric Learning for Grasping and Manipulation

This work provides an architecture that incorporates depth and tactile i...

Simple Kinesthetic Haptics for Object Recognition

Object recognition is an essential capability when performing various ta...

Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network

The ability to perceive object slip via tactile feedback enables humans ...

Deep Gated Multi-modal Learning: In-hand Object Pose Estimation with Tactile and Image

In robot manipulation tasks, especially in-hand manipulation, estimation...

E-TRoll: Tactile Sensing and Classification via A Simple Robotic Gripper for Extended Rolling Manipulations

Robotic tactile sensing provides a method of recognizing objects and the...

Please sign up or login with your details

Forgot password? Click here to reset