The CORSMAL benchmark for the prediction of the properties of containers

by   Alessio Xompero, et al.

Acoustic and visual sensing can support the contactless estimation of the weight of a container and the amount of its content when a person manipulate them. However, opaqueness and transparencies (both of the container and of the content) and the variability of materials, shapes and sizes make this problem challenging. In this paper, we present an open framework to benchmark methods for the estimation of the capacity of a container, and the type, mass, and amount of its content. The framework includes a dataset, well-defined tasks and performance measures, baselines and state-of-the-art methods, and an in-depth comparative analysis of these methods. Deep learning with neural networks with audio alone or a combination of audio and visual data are used by the methods to classify the type and amount of the content, either independently or jointly. Regression and geometric approaches with visual data are preferred to determine the capacity of the container. Results show that classifying the content type and level with methods that use only audio as input modality achieves a weighted average F1-score up to 81 Estimating the container capacity with vision-only approaches and filling mass with audio-visual, multi-stage algorithms reaches up to 65 capacity and mass scores.


page 1

page 9


Audio classification of the content of food containers and drinking glasses

Food containers, drinking glasses and cups handled by a person generate ...

Audio-Visual Object Classification for Human-Robot Collaboration

Human-robot collaboration requires the contactless estimation of the phy...

Container Localisation and Mass Estimation with an RGB-D Camera

In the research area of human-robot interactions, the automatic estimati...

Deep Audio-Visual Learning: A Survey

Audio-visual learning, aimed at exploiting the relationship between audi...

Improving Generalization of Deep Networks for Estimating Physical Properties of Containers and Fillings

We present methods to estimate the physical properties of household cont...

Top-1 CORSMAL Challenge 2020 Submission: Filling Mass Estimation Using Multi-modal Observations of Human-robot Handovers

Human-robot object handover is a key skill for the future of human-robot...

ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer

Text-to-speech(TTS) has undergone remarkable improvements in performance...

Please sign up or login with your details

Forgot password? Click here to reset