Adaptive Semantic-Visual Tree for Hierarchical Embeddings

by   Shuo Yang, et al.

Merchandise categories inherently form a semantic hierarchy with different levels of concept abstraction, especially for fine-grained categories. This hierarchy encodes rich correlations among various categories across different levels, which can effectively regularize the semantic space and thus make predictions less ambiguous. However, previous studies of fine-grained image retrieval primarily focus on semantic similarities or visual similarities. In a real application, merely using visual similarity may not satisfy the need of consumers to search merchandise with real-life images, e.g., given a red coat as a query image, we might get a red suit in recall results only based on visual similarity since they are visually similar. But the users actually want a coat rather than suit even the coat is with different color or texture attributes. We introduce this new problem based on photoshopping in real practice. That's why semantic information are integrated to regularize the margins to make "semantic" prior to "visual". To solve this new problem, we propose a hierarchical adaptive semantic-visual tree (ASVT) to depict the architecture of merchandise categories, which evaluates semantic similarities between different semantic levels and visual similarities within the same semantic class simultaneously. The semantic information satisfies the demand of consumers for similar merchandise with the query while the visual information optimizes the correlations within the semantic class. At each level, we set different margins based on the semantic hierarchy and incorporate them as prior information to learn a fine-grained feature embedding. To evaluate our framework, we propose a new dataset named JDProduct, with hierarchical labels collected from actual image queries and official merchandise images on an online shopping application. Extensive experimental results on the public CARS196 and CUB-


page 1

page 3

page 8


Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding

Object categories inherently form a hierarchy with different levels of c...

Integrating Visual and Semantic Similarity Using Hierarchies for Image Retrieval

Most of the research in content-based image retrieval (CBIR) focus on de...

Embedding Label Structures for Fine-Grained Feature Representation

Recent algorithms in convolutional neural networks (CNN) considerably ad...

Hierarchical Matching and Reasoning for Multi-Query Image Retrieval

As a promising field, Multi-Query Image Retrieval (MQIR) aims at searchi...

Hierarchical Metric Learning for Fine Grained Image Classification

This paper deals with the problem of fine-grained image classification a...

Impact of Semantic Granularity on Geographic Information Search Support

The Information Retrieval research has used semantics to provide accurat...

Semantics Preserving Hierarchy based Retrieval of Indian heritage monuments

Monument classification can be performed on the basis of their appearanc...

Please sign up or login with your details

Forgot password? Click here to reset