DeepLight: Robust Unobtrusive Real-time Screen-Camera Communication for Real-World Displays

by   Vu Tran, et al.

The paper introduces a novel, holistic approach for robust Screen-Camera Communication (SCC), where video content on a screen is visually encoded in a human-imperceptible fashion and decoded by a camera capturing images of such screen content. We first show that state-of-the-art SCC techniques have two key limitations for in-the-wild deployment: (a) the decoding accuracy drops rapidly under even modest screen extraction errors from the captured images, and (b) they generate perceptible flickers on common refresh rate screens even with minimal modulation of pixel intensity. To overcome these challenges, we introduce DeepLight, a system that incorporates machine learning (ML) models in the decoding pipeline to achieve humanly-imperceptible, moderately high SCC rates under diverse real-world conditions. Deep-Light's key innovation is the design of a Deep Neural Network (DNN) based decoder that collectively decodes all the bits spatially encoded in a display frame, without attempting to precisely isolate the pixels associated with each encoded bit. In addition, DeepLight supports imperceptible encoding by selectively modulating the intensity of only the Blue channel, and provides reasonably accurate screen extraction (IoU values >= 83 pipelines. We show that a fully functional DeepLight system is able to robustly achieve high decoding accuracy (frame error rate < 0.2) and moderately-high data goodput (>=0.95Kbps) using a human-held smartphone camera, even over larger screen-camera distances (approx =2m).


page 1

page 2

page 3

page 5

page 6

page 7

page 14


Design and Implementation of a Novel Compatible Encoding Scheme in the Time Domain for Image Sensor Communication

This paper presents a modulation scheme in the time domain based on On-O...

A Feature based Approach for Video Compression

It is a high cost problem for panoramic image stitching via image matchi...

Improved Touchless Respiratory Rate Sensing

Recently, remote respiratory rate measurement techniques gained much att...

StegaStamp: Invisible Hyperlinks in Physical Photographs

Imagine a world in which each photo, printed or digitally displayed, hid...

Light Propagation Prediction through Multimode Optical Fibers with a Deep Neural Network

This work demonstrates a computational method for predicting the light p...

LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Deep neural network (DNN) architectures have been shown to outperform tr...

MadEye: Boosting Live Video Analytics Accuracy with Adaptive Camera Configurations

Camera orientations (i.e., rotation and zoom) govern the content that a ...

Please sign up or login with your details

Forgot password? Click here to reset