Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing

by   Aditay Tripathi, et al.

This paper presents a framework for jointly grounding objects that follow certain semantic relationship constraints given in a scene graph. A typical natural scene contains several objects, often exhibiting visual relationships of varied complexities between them. These inter-object relationships provide strong contextual cues toward improving grounding performance compared to a traditional object query-only-based localization task. A scene graph is an efficient and structured way to represent all the objects and their semantic relationships in the image. In an attempt towards bridging these two modalities representing scenes and utilizing contextual information for improving object localization, we rigorously study the problem of grounding scene graphs on natural images. To this end, we propose a novel graph neural network-based approach referred to as Visio-Lingual Message PAssing Graph Neural Network (VL-MPAG Net). In VL-MPAG Net, we first construct a directed graph with object proposals as nodes and an edge between a pair of nodes representing a plausible relation between them. Then a three-step inter-graph and intra-graph message passing is performed to learn the context-dependent representation of the proposals and query objects. These object representations are used to score the proposals to generate object localization. The proposed method significantly outperforms the baselines on four public datasets.


page 1

page 3

page 7

page 12

page 13


HL-Net: Heterophily Learning Network for Scene Graph Generation

Scene graph generation (SGG) aims to detect objects and predict their pa...

Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network

Recent scene graph generation (SGG) frameworks have focused on learning ...

SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation

In this paper we propose a neural message passing approach to augment an...

Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search

Searching for objects in indoor organized environments such as homes or ...

GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering

Images are more than a collection of objects or attributes – they repres...

Suspected Object Matters: Rethinking Model's Prediction for One-stage Visual Grounding

Recently, one-stage visual grounders attract high attention due to the c...

SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans

We present a novel approach to reconstructing lightweight, CAD-based rep...

Please sign up or login with your details

Forgot password? Click here to reset