Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation

by   Manish Sahu, et al.

Purpose: Segmentation of surgical instruments in endoscopic videos is essential for automated surgical scene understanding and process modeling. However, relying on fully supervised deep learning for this task is challenging because manual annotation occupies valuable time of the clinical experts. Methods: We introduce a teacher-student learning approach that learns jointly from annotated simulation data and unlabeled real data to tackle the erroneous learning problem of the current consistency-based unsupervised domain adaptation framework. Results: Empirical results on three datasets highlight the effectiveness of the proposed framework over current approaches for the endoscopic instrument segmentation task. Additionally, we provide analysis of major factors affecting the performance on all datasets to highlight the strengths and failure modes of our approach. Conclusion: We show that our proposed approach can successfully exploit the unlabeled real endoscopic video frames and improve generalization performance over pure simulation-based training and the previous state-of-the-art. This takes us one step closer to effective segmentation of surgical tools in the annotation scarce setting.


page 2

page 9

page 11

page 12


Endo-Sim2Real: Consistency learning-based domain adaptation for instrument segmentation

Surgical tool segmentation in endoscopic videos is an important componen...

Rethinking Surgical Instrument Segmentation: A Background Image Can Be All You Need

Data diversity and volume are crucial to the success of training deep le...

Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition

Vision algorithms capable of interpreting scenes from a real-time video ...

Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis

Despite their impressive performance in various surgical scene understan...

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

Automatic surgical instrument segmentation of endoscopic images is a cru...

Exploiting the potential of unlabeled endoscopic video data with self-supervised learning

Purpose: Due to the breakthrough successes of deep learning-based soluti...

Please sign up or login with your details

Forgot password? Click here to reset