Combiner and HyperCombiner Networks: Rules to Combine Multimodality MR Images for Prostate Cancer Localisation

by   Wen Yan, et al.

One of the distinct characteristics in radiologists' reading of multiparametric prostate MR scans, using reporting systems such as PI-RADS v2.1, is to score individual types of MR modalities, T2-weighted, diffusion-weighted, and dynamic contrast-enhanced, and then combine these image-modality-specific scores using standardised decision rules to predict the likelihood of clinically significant cancer. This work aims to demonstrate that it is feasible for low-dimensional parametric models to model such decision rules in the proposed Combiner networks, without compromising the accuracy of predicting radiologic labels: First, it is shown that either a linear mixture model or a nonlinear stacking model is sufficient to model PI-RADS decision rules for localising prostate cancer. Second, parameters of these (generalised) linear models are proposed as hyperparameters, to weigh multiple networks that independently represent individual image modalities in the Combiner network training, as opposed to end-to-end modality ensemble. A HyperCombiner network is developed to train a single image segmentation network that can be conditioned on these hyperparameters during inference, for much improved efficiency. Experimental results based on data from 850 patients, for the application of automating radiologist labelling multi-parametric MR, compare the proposed combiner networks with other commonly-adopted end-to-end networks. Using the added advantages of obtaining and interpreting the modality combining rules, in terms of the linear weights or odds-ratios on individual image modalities, three clinical applications are presented for prostate cancer segmentation, including modality availability assessment, importance quantification and rule discovery.


Organ At Risk Segmentation with Multiple Modality

With the development of image segmentation in computer vision, biomedica...

Transfer Brain MRI Tumor Segmentation Models Across Modalities with Adversarial Networks

In this work, we present an approach to brain cancer segmentation in Mag...

Constructing multi-modality and multi-classifier radiomics predictive models through reliable classifier fusion

Radiomics aims to extract and analyze large numbers of quantitative feat...

A Neural Ordinary Differential Equation Model for Visualizing Deep Neural Network Behaviors in Multi-Parametric MRI based Glioma Segmentation

Purpose: To develop a neural ordinary differential equation (ODE) model ...

Cross-Modality Image Registration using a Training-Time Privileged Third Modality

In this work, we consider the task of pairwise cross-modality image regi...

Bi-parametric prostate MR image synthesis using pathology and sequence-conditioned stable diffusion

We propose an image synthesis mechanism for multi-sequence prostate MR i...

Computerized Multiparametric MR image Analysis for Prostate Cancer Aggressiveness-Assessment

We propose an automated method for detecting aggressive prostate cancer(...

Please sign up or login with your details

Forgot password? Click here to reset