Multi-granularity Association Learning Framework for on-the-fly Fine-Grained Sketch-based Image Retrieval

by   Dawei Dai, et al.

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo in a given query sketch. However, its widespread applicability is limited by the fact that it is difficult to draw a complete sketch for most people, and the drawing process often takes time. In this study, we aim to retrieve the target photo with the least number of strokes possible (incomplete sketch), named on-the-fly FG-SBIR (Bhunia et al. 2020), which starts retrieving at each stroke as soon as the drawing begins. We consider that there is a significant correlation among these incomplete sketches in the sketch drawing episode of each photo. To learn more efficient joint embedding space shared between the photo and its incomplete sketches, we propose a multi-granularity association learning framework that further optimizes the embedding space of all incomplete sketches. Specifically, based on the integrity of the sketch, we can divide a complete sketch episode into several stages, each of which corresponds to a simple linear mapping layer. Moreover, our framework guides the vector space representation of the current sketch to approximate that of its later sketches to realize the retrieval performance of the sketch with fewer strokes to approach that of the sketch with more strokes. In the experiments, we proposed more realistic challenges, and our method achieved superior early retrieval efficiency over the state-of-the-art methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.


Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the proble...

Active Learning for Fine-Grained Sketch-Based Image Retrieval

The ability to retrieve a photo by mere free-hand sketching highlights t...

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a sp...

Generic Sketch-Based Retrieval Learned without Drawing a Single Sketch

We cast the sketch-based retrieval as edge-map matching. A shared convol...

Towards Practicality of Sketch-Based Visual Understanding

Sketches have been used to conceptualise and depict visual objects from ...

Deep Factorised Inverse-Sketching

Modelling human free-hand sketches has become topical recently, driven b...

Compact and Effective Representations for Sketch-based Image Retrieval

Sketch-based image retrieval (SBIR) has undergone an increasing interest...

Please sign up or login with your details

Forgot password? Click here to reset