An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

by   Lei Zhu, et al.

Due to the rapid development of mobile Internet techniques, cloud computation and popularity of online social networking and location-based services, massive amount of multimedia data with geographical information is generated and uploaded to the Internet. In this paper, we propose a novel type of cross-modal multimedia retrieval called geo-multimedia cross-modal retrieval which aims to search out a set of geo-multimedia objects based on geographical distance proximity and semantic similarity between different modalities. Previous studies for cross-modal retrieval and spatial keyword search cannot address this problem effectively because they do not consider multimedia data with geo-tags and do not focus on this type of query. In order to address this problem efficiently, we present the definition of kNN geo-multimedia cross-modal query at the first time and introduce relevant conceptions such as cross-modal semantic representation space. To bridge the semantic gap between different modalities, we propose a method named cross-modal semantic matching which contains two important component, i.e., CorrProj and LogsTran, which aims to construct a common semantic representation space for cross-modal semantic similarity measurement. Besides, we designed a framework based on deep learning techniques to implement common semantic representation space construction. In addition, a novel hybrid indexing structure named GMR-Tree combining geo-multimedia data and R-Tree is presented and a efficient kNN search algorithm called kGMCMS is designed. Comprehensive experimental evaluation on real and synthetic dataset clearly demonstrates that our solution outperforms the-state-of-the-art methods.


page 4

page 22


Cross-Modal Learning via Pairwise Constraints

In multimedia applications, the text and image components in a web docum...

Efficient Continuous Top-k Geo-Image Search on Road Network

With the rapid development of mobile Internet and cloud computing techno...

Efficient Interactive Search for Geo-tagged Multimedia Data

Due to the advances in mobile computing and multimedia techniques, there...

Discriminative Semantic Transitive Consistency for Cross-Modal Learning

Cross-modal retrieval is generally performed by projecting and aligning ...

Performance Effectiveness of Multimedia Information Search Using the Epsilon-Greedy Algorithm

In the search and retrieval of multimedia objects, it is impractical to ...

Research on the Brain-inspired Cross-modal Neural Cognitive Computing Framework

To address modeling problems of brain-inspired intelligence, this thesis...

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Cross-modal retrieval aims to retrieve relevant data across different mo...

Please sign up or login with your details

Forgot password? Click here to reset