Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

by   Qiang Zhang, et al.

Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model, which can then be used on its own during the test time. The tactile model aims to predict the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, estimate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.


page 1

page 3

page 4

page 5

page 6


Tactile Model O: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand

Bringing tactile sensation to robotic hands will allow for more effectiv...

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Touch sensing is widely acknowledged to be important for dexterous robot...

Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images

The connection between visual input and tactile sensing is critical for ...

In-Hand Object-Dynamics Inference using Tactile Fingertips

Having the ability to estimate an object's properties through interactio...

Understanding Dynamic Tactile Sensing for Liquid Property Estimation

Humans perceive the world by interacting with objects, which often happe...

Adaptive Grasp Control through Multi-Modal Interactions for Assistive Prosthetic Devices

The hand is one of the most complex and important parts of the human bod...

Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

We investigate how high-resolution tactile sensors can be utilized in co...

Please sign up or login with your details

Forgot password? Click here to reset