Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts

by   Annie S. Chen, et al.

Effective machine learning models learn both robust features that directly determine the outcome of interest (e.g., an object with wheels is more likely to be a car), and shortcut features (e.g., an object on a road is more likely to be a car). The latter can be a source of error under distributional shift, when the correlations change at test-time. The prevailing sentiment in the robustness literature is to avoid such correlative shortcut features and learn robust predictors. However, while robust predictors perform better on worst-case distributional shifts, they often sacrifice accuracy on majority subpopulations. In this paper, we argue that shortcut features should not be entirely discarded. Instead, if we can identify the subpopulation to which an input belongs, we can adaptively choose among models with different strengths to achieve high performance on both majority and minority subpopulations. We propose COnfidence-baSed MOdel Selection (CosMoS), where we observe that model confidence can effectively guide model selection. Notably, CosMoS does not require any target labels or group annotations, either of which may be difficult to obtain or unavailable. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift. We find that CosMoS achieves 2-5 across all subpopulations, compared to using only robust predictors or other model aggregation methods.


page 1

page 2

page 3

page 4


Group Model Selection Using Marginal Correlations: The Good, the Bad and the Ugly

Group model selection is the problem of determining a small subset of gr...

Estimating Generalization under Distribution Shifts via Domain-Invariant Representations

When machine learning models are deployed on a test distribution differe...

Distributionally Robust Classifiers in Sentiment Analysis

In this paper, we propose sentiment classification models based on BERT ...

Robustness to Spurious Correlations via Human Annotations

The reliability of machine learning systems critically assumes that the ...

Invariant Adversarial Learning for Distributional Robustness

Machine learning algorithms with empirical risk minimization are vulnera...

Change is Hard: A Closer Look at Subpopulation Shift

Machine learning models often perform poorly on subgroups that are under...

Distributionally Robust Learning with Stable Adversarial Training

Machine learning algorithms with empirical risk minimization are vulnera...

Please sign up or login with your details

Forgot password? Click here to reset