3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

by   Bin Li, et al.

Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the control policy from limited surgical video clips. Specially, we first extract surgical laparoscope trajectories from unlabeled videos as the demonstrations and reconstruct the corresponding surgical scenes. To fully learn from limited motion trajectory demonstrations, we propose Shape Preserving Trajectory Augmentation (SPTA) to augment these data, and build a simulation environment that supports parallel RGB-D rendering to reinforce the RL policy for interacting with the environment efficiently. With adversarial training for IL, we obtain the laparoscope control policy based on the generated rollouts and surgical demonstrations. Extensive experiments are conducted in unseen reconstructed surgical scenes, and our method outperforms the previous IL methods, which proves the feasibility of our unified learning-based framework for laparoscope control.


page 1

page 3

page 6


Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning

High-quality and representative data is essential for both Imitation Lea...

Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning

Robotic assembly tasks involve complex and low-clearance insertion traje...

Robust Learning from Observation with Model Misspecification

Imitation learning (IL) is a popular paradigm for training policies in r...

NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control

Traditionally, reinforcement learning methods predict the next action ba...

Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception

Laparoscopic Field of View (FOV) control is one of the most fundamental ...

HandMime: Sign Language Fingerspelling Acquisition via Imitation Learning

Learning fine-grained movements is among the most challenging topics in ...

Learning and Extrapolation of Robotic Skills using Task-Parameterized Equation Learner Networks

Imitation learning approaches achieve good generalization within the ran...

Please sign up or login with your details

Forgot password? Click here to reset