Text extraction and retrieval from smartphone screenshots: Building a repository for life in media

Agnese Chiatti, Mu Jung Cho, Anupriya Gagneja, Xiao Yang, Miriam Brinberg, Katie Roehrick, Sagnik Ray Choudhury, Nilam Ram, Byron Reeves, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations


Daily engagement in life experiences is increasingly interwoven with mobile device use. Screen capture at the scale of seconds is being used in behavioral studies and to implement "just-in-time" health interventions. The increasing psychological breadth of digital information will continue to make the actual screens that people view a preferred if not required source of data about life experiences. Effective and efficient Information Extraction and Retrieval from digital screenshots is a crucial prerequisite to successful use of screen data. In this paper, we present the experimental workflow we exploited to: (i) pre-process a unique collection of screen captures, (ii) extract unstructured text embedded in the images, (iii) organize image text and metadata based on a structured schema, (iv) index the resulting document collection, and (v) allow for Image Retrieval through a dedicated vertical search engine application. The adopted procedure integrates different open source libraries for traditional image processing, Optical Character Recognition (OCR), and Image Retrieval. Our aim is to assess whether and how state-of-the-art methodologies can be applied to this novel data set. We show how combining OpenCV-based pre-processing modules with a Long short-term memory (LSTM) based release of Tesseract OCR, without ad hoc training, led to a 74% character-level accuracy of the extracted text. Further, we used the processed repository as baseline for a dedicated Image Retrieval system, for the immediate use and application for behavioral and prevention scientists. We discuss issues of Text Information Extraction and Retrieval that are particular to the screenshot image case and suggest important future work.

Original languageEnglish (US)
Title of host publicationProceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018
PublisherAssociation for Computing Machinery
Number of pages8
ISBN (Electronic)9781450351911
StatePublished - Apr 9 2018
Event33rd Annual ACM Symposium on Applied Computing, SAC 2018 - Pau, France
Duration: Apr 9 2018Apr 13 2018

Publication series

NameProceedings of the ACM Symposium on Applied Computing


Other33rd Annual ACM Symposium on Applied Computing, SAC 2018

All Science Journal Classification (ASJC) codes

  • Software


Dive into the research topics of 'Text extraction and retrieval from smartphone screenshots: Building a repository for life in media'. Together they form a unique fingerprint.

Cite this