Revisiting Hidden Representations in Transfer Learning for Medical Imaging

by   Dovile Juodelyte, et al.

While a key component to the success of deep learning is the availability of massive amounts of training data, medical image datasets are often limited in diversity and size. Transfer learning has the potential to bridge the gap between related yet different domains. For medical applications, however, it remains unclear whether it is more beneficial to pre-train on natural or medical images. We aim to shed light on this problem by comparing initialization on ImageNet and RadImageNet on seven medical classification tasks. We investigate their learned representations with Canonical Correlation Analysis (CCA) and compare the predictions of the different models. We find that overall the models pre-trained on ImageNet outperform those trained on RadImageNet. Our results show that, contrary to intuition, ImageNet and RadImageNet converge to distinct intermediate representations, and that these representations are even more dissimilar after fine-tuning. Despite these distinct representations, the predictions of the models remain similar. Our findings challenge the notion that transfer learning is effective due to the reuse of general features in the early layers of a convolutional neural network and show that weight similarity before and after fine-tuning is negatively related to performance gains.


Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning

Remarkable progress has been made in image recognition, primarily due to...

Training Deep Learning models with small datasets

The growing use of Machine Learning has produced significant advances in...

Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes

Deep learning is the current bet for image classification. Its greed for...

Bridging the gap between Natural and Medical Images through Deep Colorization

Deep learning has thrived by training on large-scale datasets. However, ...

Transfusion: Understanding Transfer Learning with Applications to Medical Imaging

With the increasingly varied applications of deep learning, transfer lea...

When How to Transfer with Transfer Learning

In deep learning, transfer learning (TL) has become the de facto approac...

Please sign up or login with your details

Forgot password? Click here to reset