Reproducibility of the Methods in Medical Imaging with Deep Learning

by   Attila Simkó, et al.

Concerns about the reproducibility of deep learning research are more prominent than ever, with no clear solution in sight. The relevance of machine learning research can only be improved if we also employ empirical rigor that incorporates reproducibility guidelines, especially so in the medical imaging field. The Medical Imaging with Deep Learning (MIDL) conference has made advancements in this direction by advocating open access, and recently also recommending authors to make their code public - both aspects being adopted by the majority of the conference submissions. This helps the reproducibility of the methods, however, there is currently little or no support for further evaluation of these supplementary material, making them vulnerable to poor quality, which affects the impact of the entire submission. We have evaluated all accepted full paper submissions to MIDL between 2018 and 2022 using established, but slightly adjusted guidelines on reproducibility and the quality of the public repositories. The evaluations show that publishing repositories and using public datasets are becoming more popular, which helps traceability, but the quality of the repositories has not improved over the years, leaving room for improvement in every aspect of designing repositories. Merely 22 using our evaluations. From the commonly encountered issues during the evaluations, we propose a set of guidelines for machine learning-related research for medical imaging applications, adjusted specifically for future submissions to MIDL.


page 1

page 2

page 3

page 4


Reproducibility in machine learning for medical imaging

Reproducibility is a cornerstone of science, as the replication of findi...

Open-radiomics: A Research Protocol to Make Radiomics-based Machine Learning Pipelines Reproducible

The application of artificial intelligence (AI) techniques to medical im...

How I failed machine learning in medical imaging – shortcomings and recommendations

Medical imaging is an important research field with many opportunities f...

The use of deep learning in interventional radiotherapy (brachytherapy): a review with a focus on open source and open data

Deep learning advanced to one of the most important technologies in almo...

Reproducibility in medical image radiomic studies: contribution of dynamic histogram binning

The de facto standard of dynamic histogram binning for radiomic feature ...

Systematic Review of Techniques in Brain Image Synthesis using Deep Learning

This review paper delves into the present state of medical imaging, with...

Guaranteeing Reproducibility in Deep Learning Competitions

To encourage the development of methods with reproducible and robust tra...

Please sign up or login with your details

Forgot password? Click here to reset