A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

by   Rema Daher, et al.

Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames where they are not present in the same location. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner that relies on automatic detection of specular highlights. System evaluations show significant improvements to traditional methods through direct comparison as well as other machine learning techniques through an ablation study that depicts the importance of the network's temporal and transfer learning components. The generalizability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of specular highlight inpainting on these tasks in a novel comprehensive analysis.


page 1

page 6

page 8

page 9

page 10

page 11

page 12


Frame-Recurrent Video Inpainting by Robust Optical Flow Inference

In this paper, we present a new inpainting framework for recovering miss...

Optical Flow Estimation from a Single Motion-blurred Image

In most of computer vision applications, motion blur is regarded as an u...

Faster Unsupervised Semantic Inpainting: A GAN Based Approach

In this paper, we propose to improve the inference speed and visual qual...

TransFlow: Transformer as Flow Learner

Optical flow is an indispensable building block for various important co...

Deep Generative Filter for Motion Deblurring

Removing blur caused by camera shake in images has always been a challen...

Improved Techniques for GAN based Facial Inpainting

In this paper we present several architectural and optimization recipes ...

Domain aware medical image classifier interpretation by counterfactual impact analysis

The success of machine learning methods for computer vision tasks has dr...

Please sign up or login with your details

Forgot password? Click here to reset