Towards Consistent Predictive Confidence through Fitted Ensembles

by   Navid Kardan, et al.

Deep neural networks are behind many of the recent successes in machine learning applications. However, these models can produce overconfident decisions while encountering out-of-distribution (OOD) examples or making a wrong prediction. This inconsistent predictive confidence limits the integration of independently-trained learning models into a larger system. This paper introduces separable concept learning framework to realistically measure the performance of classifiers in presence of OOD examples. In this setup, several instances of a classifier are trained on different parts of a partition of the set of classes. Later, the performance of the combination of these models is evaluated on a separate test set. Unlike current OOD detection techniques, this framework does not require auxiliary OOD datasets and does not separate classification from detection performance. Furthermore, we present a new strong baseline for more consistent predictive confidence in deep models, called fitted ensembles, where overconfident predictions are rectified by transformed versions of the original classification task. Fitted ensembles can naturally detect OOD examples without requiring auxiliary data by observing contradicting predictions among its components. Experiments on MNIST, SVHN, CIFAR-10/100, and ImageNet show fitted ensemble significantly outperform conventional ensembles on OOD examples and are possible to scale.


page 1

page 2

page 3

page 4


Pathologies of Predictive Diversity in Deep Ensembles

Classical results establish that ensembles of small models benefit when ...

Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection

Machine learning models are often used in practice if they achieve good ...

Learning Confidence for Out-of-Distribution Detection in Neural Networks

Modern neural networks are very powerful predictive models, but they are...

Selective Ensembles for Consistent Predictions

Recent work has shown that models trained to the same objective, and whi...

Relevant-features based Auxiliary Cells for Energy Efficient Detection of Natural Errors

Deep neural networks have demonstrated state-of-the-art performance on m...

Fitted Learning: Models with Awareness of their Limits

Though deep learning has pushed the boundaries of classification forward...

Estimating Confidence of Predictions of Individual Classifiers and Their Ensembles for the Genre Classification Task

Genre identification is a subclass of non-topical text classification. T...

Please sign up or login with your details

Forgot password? Click here to reset