UbiPhysio: Support Daily Functioning, Fitness, and Rehabilitation with Action Understanding and Feedback in Natural Language

by   Chongyang Wang, et al.

We introduce UbiPhysio, a milestone framework that delivers fine-grained action description and feedback in natural language to support people's daily functioning, fitness, and rehabilitation activities. This expert-like capability assists users in properly executing actions and maintaining engagement in remote fitness and rehabilitation programs. Specifically, the proposed UbiPhysio framework comprises a fine-grained action descriptor and a knowledge retrieval-enhanced feedback module. The action descriptor translates action data, represented by a set of biomechanical movement features we designed based on clinical priors, into textual descriptions of action types and potential movement patterns. Building on physiotherapeutic domain knowledge, the feedback module provides clear and engaging expert feedback. We evaluated UbiPhysio's performance through extensive experiments with data from 104 diverse participants, collected in a home-like setting during 25 types of everyday activities and exercises. We assessed the quality of the language output under different tuning strategies using standard benchmarks. We conducted a user study to gather insights from clinical experts and potential users on our framework. Our initial tests show promise for deploying UbiPhysio in real-life settings without specialized devices.


page 9

page 10

page 17

page 19


Weakly-Supervised Temporal Action Detection for Fine-Grained Videos with Hierarchical Atomic Actions

Action understanding has evolved into the era of fine granularity, as mo...

The Curious Layperson: Fine-Grained Image Recognition without Expert Labels

Most of us are not experts in specific fields, such as ornithology. None...

FLAG3D: A 3D Fitness Activity Dataset with Language Instruction

With the continuously thriving popularity around the world, fitness acti...

Inferring Temporal Compositions of Actions Using Probabilistic Automata

This paper presents a framework to recognize temporal compositions of at...

Fine-grained Action Segmentation using the Semi-Supervised Action GAN

In this paper we address the problem of continuous fine-grained action s...

TRACE: Transform Aggregate and Compose Visiolinguistic Representations for Image Search with Text Feedback

The ability to efficiently search for images over an indexed database is...

Exploiting Fine-Grained DCT Representations for Hiding Image-Level Messages within JPEG Images

Unlike hiding bit-level messages, hiding image-level messages is more ch...

Please sign up or login with your details

Forgot password? Click here to reset