Trackerless freehand ultrasound with sequence modelling and auxiliary transformation over past and future frames

by   Qi Li, et al.

Three-dimensional (3D) freehand ultrasound (US) reconstruction without a tracker can be advantageous over its two-dimensional or tracked counterparts in many clinical applications. In this paper, we propose to estimate 3D spatial transformation between US frames from both past and future 2D images, using feed-forward and recurrent neural networks (RNNs). With the temporally available frames, a further multi-task learning algorithm is proposed to utilise a large number of auxiliary transformation-predicting tasks between them. Using more than 40,000 US frames acquired from 228 scans on 38 forearms of 19 volunteers in a volunteer study, the hold-out test performance is quantified by frame prediction accuracy, volume reconstruction overlap, accumulated tracking error and final drift, based on ground-truth from an optical tracker. The results show the importance of modelling the temporal-spatially correlated input frames as well as output transformations, with further improvement owing to additional past and/or future frames. The best performing model was associated with predicting transformation between moderately-spaced frames, with an interval of less than ten frames at 20 frames per second (fps). Little benefit was observed by adding frames more than one second away from the predicted transformation, with or without LSTM-based RNNs. Interestingly, with the proposed approach, explicit within-sequence loss that encourages consistency in composing transformations or minimises accumulated error may no longer be required. The implementation code and volunteer data will be made publicly available ensuring reproducibility and further research.


page 1

page 2

page 3

page 4


Transformation-Based Models of Video Sequences

In this work we propose a simple unsupervised approach for next frame pr...

Predicting tongue motion in unlabeled ultrasound videos using convolutional LSTM neural network

A challenge in speech production research is to predict future tongue mo...

Privileged Anatomical and Protocol Discrimination in Trackerless 3D Ultrasound Reconstruction

Three-dimensional (3D) freehand ultrasound (US) reconstruction without u...

Predicting the Future with Transformational States

An intelligent observer looks at the world and sees not only what is, bu...

ImplicitVol: Sensorless 3D Ultrasound Reconstruction with Deep Implicit Representation

The objective of this work is to achieve sensorless reconstruction of a ...

Temporally Robust Global Motion Compensation by Keypoint-based Congealing

Global motion compensation (GMC) removes the impact of camera motion and...

Modeling sequential data using higher-order relational features and predictive training

Bi-linear feature learning models, like the gated autoencoder, were prop...

Please sign up or login with your details

Forgot password? Click here to reset