Challenges in Annotation of useR Data for UbiquitOUs Systems: Results from the 1st ARDUOUS Workshop

by   Kristina Yordanova, et al.

Labelling user data is a central part of the design and evaluation of pervasive systems that aim to support the user through situation-aware reasoning. It is essential both in designing and training the system to recognise and reason about the situation, either through the definition of a suitable situation model in knowledge-driven applications, or through the preparation of training data for learning tasks in data-driven models. Hence, the quality of annotations can have a significant impact on the performance of the derived systems. Labelling is also vital for validating and quantifying the performance of applications. In particular, comparative evaluations require the production of benchmark datasets based on high-quality and consistent annotations. With pervasive systems relying increasingly on large datasets for designing and testing models of users' activities, the process of data labelling is becoming a major concern for the community. In this work we present a qualitative and quantitative analysis of the challenges associated with annotation of user data and possible strategies towards addressing these challenges. The analysis was based on the data gathered during the 1st International Workshop on Annotation of useR Data for UbiquitOUs Systems (ARDUOUS) and consisted of brainstorming as well as annotation and questionnaire data gathered during the talks, poster session, live annotation session, and discussion session.


page 7

page 9

page 10

page 11

page 12


Proceedings of the 12th Workshop on User Interfaces for Theorem Provers

The UITP workshop series brings together researchers interested in desig...

Block Annotation: Better Image Annotation for Semantic Segmentation with Sub-Image Decomposition

Image datasets with high-quality pixel-level annotations are valuable fo...

VerSe: A Vertebrae Labelling and Segmentation Benchmark

In this paper we report the challenge set-up and results of the Large Sc...

A survey of image labelling for computer vision applications

Supervised machine learning methods for image analysis require large amo...

NEAL: An open-source tool for audio annotation

Passive acoustic monitoring is used widely in ecology, biodiversity, and...

Data Processing and Annotation Schemes for FinCausal Shared Task

This document explains the annotation schemes used to label the data for...

Creating user stereotypes for persona development from qualitative data through semi-automatic subspace clustering

Personas are models of users that incorporate motivations, wishes, and o...

Please sign up or login with your details

Forgot password? Click here to reset