Practicality of generalization guarantees for unsupervised domain adaptation with neural networks

by   Adam Breitholtz, et al.

Understanding generalization is crucial to confidently engineer and deploy machine learning models, especially when deployment implies a shift in the data domain. For such domain adaptation problems, we seek generalization bounds which are tractably computable and tight. If these desiderata can be reached, the bounds can serve as guarantees for adequate performance in deployment. However, in applications where deep neural networks are the models of choice, deriving results which fulfill these remains an unresolved challenge; most existing bounds are either vacuous or has non-estimable terms, even in favorable conditions. In this work, we evaluate existing bounds from the literature with potential to satisfy our desiderata on domain adaptation image classification tasks, where deep neural networks are preferred. We find that all bounds are vacuous and that sample generalization terms account for much of the observed looseness, especially when these terms interact with measures of domain shift. To overcome this and arrive at the tightest possible results, we combine each bound with recent data-dependent PAC-Bayes analysis, greatly improving the guarantees. We find that, when domain overlap can be assumed, a simple importance weighting extension of previous work provides the tightest estimable bound. Finally, we study which terms dominate the bounds and identify possible directions for further improvement.


page 1

page 2

page 3

page 4


Unsupervised Domain Adaptation with Deep Neural-Network

This report contributes to the field of unsupervised domain adaptation b...

An Improvement to the Domain Adaptation Bound in a PAC-Bayesian context

This paper provides a theoretical analysis of domain adaptation based on...

Learning Weighted Representations for Generalization Across Designs

Predictive models that generalize well under distributional shift are of...

Domain Discrepancy Measure Using Complex Models in Unsupervised Domain Adaptation

We address the problem of measuring the difference between two domains i...

PAC Learning Guarantees Under Covariate Shift

We consider the Domain Adaptation problem, also known as the covariate s...

Neural Unsupervised Domain Adaptation in NLP—A Survey

Deep neural networks excel at learning from labeled data and achieve sta...

Context-Adaptive Deep Neural Networks via Bridge-Mode Connectivity

The deployment of machine learning models in safety-critical application...

Please sign up or login with your details

Forgot password? Click here to reset