Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigation

by   Gyan Tatiya, et al.

Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from a series of pre-training tasks – all within a reinforcement learning framework for audio-visual navigation. We also define a new audio-visual navigation sub-task, where agents are evaluated on novel sounding objects, as opposed to unheard clips of known objects. We show improvements over strong baselines in generalisation to unseen regions and novel sounding objects, within the Habitat-Matterport3D simulation environment, under the SoundSpaces task.


page 2

page 9

page 17

page 18

page 19


Visual Semantic Navigation using Scene Priors

How do humans navigate to target objects in novel scenes? Do we use the ...

Semantic Audio-Visual Navigation

Recent work on audio-visual navigation assumes a constantly-sounding tar...

Learning Embeddings that Capture Spatial Semantics for Indoor Navigation

Incorporating domain-specific priors in search and navigation tasks has ...

Pathdreamer: A World Model for Indoor Navigation

People navigating in unfamiliar buildings take advantage of myriad visua...

Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation

The task of Visual Object Navigation (VON) involves an agent's ability t...

Object-oriented Targets for Visual Navigation using Rich Semantic Representations

When searching for an object humans navigate through a scene using seman...

Generalization to Novel Objects using Prior Relational Knowledge

To solve tasks in new environments involving objects unseen during train...

Please sign up or login with your details

Forgot password? Click here to reset