CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation

by   Dogan Altan, et al.

Robot safety has been a prominent research topic in recent years since robots are more involved in daily tasks. It is crucial to devise the required safety mechanisms to enable service robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during the execution of these tasks. Detection and identification of these anomalies is an essential step towards fulfilling these requirements. Although several architectures are proposed for anomaly detection, identification is not yet thoroughly investigated. This task is challenging since indicators may appear long before anomalies are detected. In this paper, we propose a ConvoLUtional threE-stream Anomaly Identification (CLUE-AI) framework to address this problem. The framework fuses visual, auditory and proprioceptive data streams to identify everyday object manipulation anomalies. A stream of 2D images gathered through an RGB-D camera placed on the head of the robot is processed within a self-attention enabled visual stage to capture visual anomaly indicators. The auditory modality provided by the microphone placed on the robot's lower torso is processed within a designed convolutional neural network (CNN) in the auditory stage. Last, the force applied by the gripper and the gripper state are processed within a CNN to obtain proprioceptive features. These outputs are then combined with a late fusion scheme. Our novel three-stream framework design is analyzed on everyday object manipulation tasks with a Baxter humanoid robot in a semi-structured setting. The results indicate that the framework achieves an f-score of 94 in classifying anomalies that arise during runtime.


page 9

page 16

page 17


What went wrong?: Identification of Everyday Object Manipulation Anomalies

Extending the abilities of service robots is important for expanding wha...

Challenges in Visual Anomaly Detection for Mobile Robots

We consider the task of detecting anomalies for autonomous mobile robots...

Sensing Anomalies as Potential Hazards: Datasets and Benchmarks

We consider the problem of detecting, in the visual sensing data stream ...

Using Visual Anomaly Detection for Task Execution Monitoring

Execution monitoring is essential for robots to detect and respond to fa...

Self-supervised Representation Learning for Reliable Robotic Monitoring of Fruit Anomalies

Data augmentation can be a simple yet powerful tool for autonomous robot...

Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream

Manipulation tasks in daily life, such as pouring water, unfold intentio...

Please sign up or login with your details

Forgot password? Click here to reset