HOSE-Net: Higher Order Structure Embedded Network for Scene Graph Generation

by   Meng Wei, et al.

Scene graph generation aims to produce structured representations for images, which requires to understand the relations between objects. Due to the continuous nature of deep neural networks, the prediction of scene graphs is divided into object detection and relation classification. However, the independent relation classes cannot separate the visual features well. Although some methods organize the visual features into graph structures and use message passing to learn contextual information, they still suffer from drastic intra-class variations and unbalanced data distributions. One important factor is that they learn an unstructured output space that ignores the inherent structures of scene graphs. Accordingly, in this paper, we propose a Higher Order Structure Embedded Network (HOSE-Net) to mitigate this issue. First, we propose a novel structure-aware embedding-to-classifier(SEC) module to incorporate both local and global structural information of relationships into the output space. Specifically, a set of context embeddings are learned via local graph based message passing and then mapped to a global structure based classification space. Second, since learning too many context-specific classification subspaces can suffer from data sparsity issues, we propose a hierarchical semantic aggregation(HSA) module to reduces the number of subspaces by introducing higher order structural information. HSA is also a fast and flexible tool to automatically search a semantic object hierarchy based on relational knowledge graphs. Extensive experiments show that the proposed HOSE-Net achieves the state-of-the-art performance on two popular benchmarks of Visual Genome and VRD.


page 1

page 2

page 3

page 4

page 5

page 7

page 8

page 9


HL-Net: Heterophily Learning Network for Scene Graph Generation

Scene graph generation (SGG) aims to detect objects and predict their pa...

Attentive Relational Networks for Mapping Images to Scene Graphs

Scene graph generation refers to the task of automatically mapping an im...

Prototype-based Embedding Network for Scene Graph Generation

Current Scene Graph Generation (SGG) methods explore contextual informat...

High Order Structure Descriptors for Scene Images

Structure information is ubiquitous in natural scene images and it plays...

Neural Motifs: Scene Graph Parsing with Global Context

We investigate the problem of producing structured graph representations...

Learning to Compose Dynamic Tree Structures for Visual Contexts

We propose to compose dynamic tree structures that place the objects in ...

Neural Belief Propagation for Scene Graph Generation

Scene graph generation aims to interpret an input image by explicitly mo...

Please sign up or login with your details

Forgot password? Click here to reset