Automatic Visual Theme Discovery from Joint Image and Text Corpora

by   Ke Sun, et al.

A popular approach to semantic image understanding is to manually tag images with keywords and then learn a mapping from vi- sual features to keywords. Manually tagging images is a subjective pro- cess and the same or very similar visual contents are often tagged with different keywords. Furthermore, not all tags have the same descriptive power for visual contents and large vocabulary available from natural language could result in a very diverse set of keywords. In this paper, we propose an unsupervised visual theme discovery framework as a better (more compact, efficient and effective) alternative to semantic represen- tation of visual contents. We first show that tag based annotation lacks consistency and compactness for describing visually similar contents. We then learn the visual similarity between tags based on the visual features of the images containing the tags. At the same time, we use a natural language processing technique (word embedding) to measure the seman- tic similarity between tags. Finally, we cluster tags into visual themes based on their visual similarity and semantic similarity measures using a spectral clustering algorithm. We conduct user studies to evaluate the effectiveness and rationality of the visual themes discovered by our unsu- pervised algorithm and obtains promising result. We then design three common computer vision tasks, example based image search, keyword based image search and image labelling to explore potential applica- tion of our visual themes discovery framework. In experiments, visual themes significantly outperforms tags on semantic image understand- ing and achieve state-of-art performance in all three tasks. This again demonstrate the effectiveness and versatility of proposed framework.


page 1

page 2

page 3

page 4


A Multi-View Embedding Space for Modeling Internet Images, Tags, and their Semantics

This paper investigates the problem of modeling Internet images and asso...

Finding the Topic of a Set of Images

In this paper we introduce the problem of determining the topic that a s...

Analysing Word Importance for Image Annotation

Image annotation provides several keywords automatically for a given ima...

Tagging like Humans: Diverse and Distinct Image Annotation

In this work we propose a new automatic image annotation model, dubbed ...

On- Device Information Extraction from Screenshots in form of tags

We propose a method to make mobile screenshots easily searchable. In thi...

Label-Specific Training Set Construction from Web Resource for Image Annotation

Recently many research efforts have been devoted to image annotation by ...

Contextual Visual Similarity

Measuring visual similarity is critical for image understanding. But wha...

Please sign up or login with your details

Forgot password? Click here to reset