UDApter – Efficient Domain Adaptation Using Adapters

by   Bhavitvya Malik, et al.
National University of Singapore
Singapore University of Technology and Design

We propose two methods to make unsupervised domain adaptation (UDA) more parameter efficient using adapters, small bottleneck layers interspersed with every layer of the large-scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain-invariant information to learn task representations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Compared to strong baselines, our simple methods perform well in natural language inference (MNLI) and the cross-domain sentiment classification task. We even outperform unsupervised domain adaptation methods such as DANN and DSN in sentiment classification, and we are within 0.85 fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter


page 2

page 7


Knowledge Distillation for BERT Unsupervised Domain Adaptation

A pre-trained language model, BERT, has brought significant performance ...

ReMask: A Robust Information-Masking Approach for Domain Counterfactual Generation

Domain shift is a big challenge in NLP, thus, many approaches resort to ...

Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy

Reducing feature redundancy has shown beneficial effects for improving t...

Unsupervised Video Domain Adaptation: A Disentanglement Perspective

Unsupervised video domain adaptation is a practical yet challenging task...

Unsupervised Reinforcement Adaptation for Class-Imbalanced Text Classification

Class imbalance naturally exists when train and test models in different...

Domain Adaptation via Prompt Learning

Unsupervised domain adaption (UDA) aims to adapt models learned from a w...

Gradually Vanishing Bridge for Adversarial Domain Adaptation

In unsupervised domain adaptation, rich domain-specific characteristics ...

Please sign up or login with your details

Forgot password? Click here to reset