Text Promptable Surgical Instrument Segmentation with Vision-Language Models

by   Zijian Zhou, et al.
King's College London

In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on EndoVis2017 and EndoVis2018 datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery.


page 2

page 4

page 8

page 15

page 16

page 17

page 18


ISINet: An Instance-Based Approach for Surgical Instrument Segmentation

We study the task of semantic segmentation of surgical instruments in ro...

Self-Supervised Surgical Instrument 3D Reconstruction from a Single Camera Image

Surgical instrument tracking is an active research area that can provide...

Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries

Accurate segmentation of surgical instrument tip is an important task fo...

Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis

Despite their impressive performance in various surgical scene understan...

Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video

Deep learning-based methods have achieved promising results on surgical ...

Design and implementation of a wireless instrument adapter

The evaluation of new methods for control and manipulation in minimally ...

Please sign up or login with your details

Forgot password? Click here to reset