Kite: Automatic speech recognition for unmanned aerial vehicles

by   Dan Oneata, et al.

This paper addresses the problem of building a speech recognition system attuned to the control of unmanned aerial vehicles (UAVs). Even though UAVs are becoming widespread, the task of creating voice interfaces for them is largely unaddressed. To this end, we introduce a multi-modal evaluation dataset for UAV control, consisting of spoken commands and associated images, which represent the visual context of what the UAV "sees" when the pilot utters the command. We provide baseline results and address two research directions: (i) how robust the language models are, given an incomplete list of commands at train time; (ii) how to incorporate visual information in the language model. We find that recurrent neural networks (RNNs) are a solution to both tasks: they can be successfully adapted using a small number of commands and they can be extended to use visual cues. Our results show that the image-based RNN outperforms its text-only counterpart even if the command-image training associations are automatically generated and inherently imperfect. The dataset and our code are available at


page 1

page 4


Vision-Based Distributed Formation Control of Unmanned Aerial Vehicles

We present a novel control strategy for a team of unmanned aerial vehicl...

Automatic Crack Detection in Built Infrastructure Using Unmanned Aerial Vehicles

This paper addresses the problem of crack detection which is essential f...

Unmanned Aerial Vehicle Control Through Domain-based Automatic Speech Recognition

Currently, unmanned aerial vehicles, such as drones, are becoming a part...

How hard is it to cross the room? -- Training (Recurrent) Neural Networks to steer a UAV

This work explores the feasibility of steering a drone with a (recurrent...

Learning to Assist and Communicate with Novice Drone Pilots for Expert Level Performance

Multi-task missions for unmanned aerial vehicles (UAVs) involving inspec...

AgriColMap: Aerial-Ground Collaborative 3D Mapping for Precision Farming

The combination of aerial survey capabilities of Unmanned Aerial Vehicle...

Please sign up or login with your details

Forgot password? Click here to reset