Deep Semantic Parsing of Freehand Sketches with Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning

by   Ying Zheng, et al.

In this paper, we propose a novel deep framework for part-level semantic parsing of freehand sketches, which makes three main contributions that are experimentally shown to have substantial practical merit. First, we introduce a new idea named homogeneous transformation to address the problem of domain adaptation. For the task of sketch parsing, there is no available data of labeled freehand sketches that can be directly used for model training. An alternative solution is to learn from the existing parsing data of real images, while the domain adaptation is an inevitable problem. Unlike existing methods that utilize the edge maps of real images to approximate freehand sketches, the proposed homogeneous transformation method transforms the data from two different domains into a homogeneous space to minimize the semantic gap. Second, we design a soft-weighted loss function as guidance for the training process, which gives attention to both the ambiguous label boundary and class imbalance. Third, we present a staged learning strategy to improve the parsing performance of the trained model, which takes advantage of the shared information and specific characteristic from different sketch categories. Extensive experimental results demonstrate the effectiveness of these methods. Specifically, to evaluate the generalization ability of our homogeneous transformation method, additional experiments at the task of sketch-based image retrieval are conducted on the QMUL FG-SBIR dataset. By integrating the proposed three methods into a unified framework, our final deep semantic sketch parsing (DeepSSP) model achieves the state-of-the-art on the public SketchParse dataset.


page 1

page 3

page 8

page 9


BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR

The efficacy of zero-shot sketch-based image retrieval (ZS-SBIR) models ...

Sketch-Specific Data Augmentation for Freehand Sketch Recognition

Sketch recognition remains a significant challenge due to the limited tr...

Unsupervised Domain Adaptation with Adversarial Residual Transform Networks

Domain adaptation is widely used in learning problems lacking labels. Re...

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a sp...

A Sketch-Based System for Semantic Parsing

This paper presents our semantic parsing system for the evaluation task ...

Consistency Regularization for Generalizable Source-free Domain Adaptation

Source-free domain adaptation (SFDA) aims to adapt a well-trained source...

Plot2API: Recommending Graphic API from Plot via Semantic Parsing Guided Neural Network

Plot-based Graphic API recommendation (Plot2API) is an unstudied but mea...

Please sign up or login with your details

Forgot password? Click here to reset