Text Extraction and Retrieval from Smartphone Screenshots: Building a Repository for Life in Media

by   Agnese Chiatti, et al.

Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74 text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.


page 3

page 4

page 6

page 7


Information Retrieval from the Digitized Books

Extracting the relevant information out of a large number of documents i...

Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval

Current text-image approaches (e.g., CLIP) typically adopt dual-encoder ...

Information Extraction from Scanned Invoice Images using Text Analysis and Layout Features

While storing invoice content as metadata to avoid paper document proces...

From Unstructured to Structured: Transforming Chatbot Dialogues into Data Mart Schema for Visualization

Schools are among the primary avenues for public healthcare intervention...

A Benchmark on Tricks for Large-scale Image Retrieval

Many studies have been performed on metric learning, which has become a ...

An Automatic Image Content Retrieval Method for better Mobile Device Display User Experiences

A growing number of commercially available mobile phones come with integ...

Please sign up or login with your details

Forgot password? Click here to reset