HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation

08/12/2020
by   Meng Wei, et al.
15

Scene graph generation aims to produce structured representations for images, which requires to understand the relations between objects. Due to the continuous nature of deep neural networks, the prediction of scene graphs is divided into object detection and relation classification. However, the independent relation classes cannot separate the visual features well. Although some methods organize the visual features into graph structures and use message passing to learn contextual information, they still suffer from drastic intra-class variations and unbalanced data distributions. One important factor is that they learn an unstructured output space that ignores the inherent structures of scene graphs. Accordingly, in this paper, we propose a Higher Order Structure Embedded Network (HOSE-Net) to mitigate this issue. First, we propose a novel structure-aware embedding-to-classifier(SEC) module to incorporate both local and global structural information of relationships into the output space. Specifically, a set of context embeddings are learned via local graph based message passing and then mapped to a global structure based classification space. Second, since learning too many context-specific classification subspaces can suffer from data sparsity issues, we propose a hierarchical semantic aggregation(HSA) module to reduces the number of subspaces by introducing higher order structural information. HSA is also a fast and flexible tool to automatically search a semantic object hierarchy based on relational knowledge graphs. Extensive experiments show that the proposed HOSE-Net achieves the state-of-the-art performance on two popular benchmarks of Visual Genome and VRD.

READ FULL TEXT

page 1

page 2

page 3

page 4

page 5

page 7

page 8

page 9

05/03/2022

HL-Net: Heterophily Learning Network for Scene Graph Generation

Scene graph generation (SGG) aims to detect objects and predict their pa...
11/26/2018

Attentive Relational Networks for Mapping Images to Scene Graphs

Scene graph generation refers to the task of automatically mapping an im...
03/13/2023

Prototype-based Embedding Network for Scene Graph Generation

Current Scene Graph Generation (SGG) methods explore contextual informat...
10/15/2014

High Order Structure Descriptors for Scene Images

Structure information is ubiquitous in natural scene images and it plays...
11/17/2017

Neural Motifs: Scene Graph Parsing with Global Context

We investigate the problem of producing structured graph representations...
12/05/2018

Learning to Compose Dynamic Tree Structures for Visual Contexts

We propose to compose dynamic tree structures that place the objects in ...
12/10/2021

Neural Belief Propagation for Scene Graph Generation

Scene graph generation aims to interpret an input image by explicitly mo...

Please sign up or login with your details

Forgot password? Click here to reset