Learning to Assist and Communicate with Novice Drone Pilots for Expert Level Performance

by   Kal Backman, et al.

Multi-task missions for unmanned aerial vehicles (UAVs) involving inspection and landing tasks are challenging for novice pilots due to the difficulties associated with depth perception and the control interface. We propose a shared autonomy system, alongside supplementary information displays, to assist pilots to successfully complete multi-task missions without any pilot training. Our approach comprises of three modules: (1) a perception module that encodes visual information onto a latent representation, (2) a policy module that augments pilot's actions, and (3) an information augmentation module that provides additional information to the pilot. The policy module is trained in simulation with simulated users and transferred to the real world without modification in a user study (n=29), alongside supplementary information schemes including learnt red/green light feedback cues and an augmented reality display. The pilot's intent is unknown to the policy module and is inferred from the pilot's input and UAV's states. The assistant increased task success rate for the landing and inspection tasks from [16.67 to [95.59 performance to experienced pilots. Red/green light feedback cues reduced the required time by 19.53 task, where participants rated it as their preferred condition due to the intuitive interface and providing reassurance. This work demonstrates that simple user models can train shared autonomy systems in simulation, and transfer to physical tasks to estimate user intent and provide effective assistance and information to the pilot.


page 1

page 3

page 6

page 7

page 12

page 13

page 15

page 16


Reinforcement Learning for Shared Autonomy Drone Landings

Novice pilots find it difficult to operate and land unmanned aerial vehi...

Learning to Assist Drone Landings

Unmanned aerial vehicles (UAVs) are often used for navigating dangerous ...

A Virtual Reality-based Training and Assessment System for Bridge Inspectors with an Assistant Drone

Over 600,000 bridges in the U.S. must be inspected every two years to id...

Kite: Automatic speech recognition for unmanned aerial vehicles

This paper addresses the problem of building a speech recognition system...

X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback

We aim to help users communicate their intent to machines using flexible...

In-Device Feedback in Immersive Head-Mounted Displays for Distance Perception During Teleoperation of Unmanned Ground Vehicles

In recent years, Virtual Reality (VR) Head-Mounted Displays (HMD) have b...

Assisted Perception: Optimizing Observations to Communicate State

We aim to help users estimate the state of the world in tasks like robot...

Please sign up or login with your details

Forgot password? Click here to reset