UDApter – Efficient Domain Adaptation Using Adapters

02/07/2023
by   Bhavitvya Malik, et al.
National University of Singapore
Singapore University of Technology and Design
0

We propose two methods to make unsupervised domain adaptation (UDA) more parameter efficient using adapters, small bottleneck layers interspersed with every layer of the large-scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain-invariant information to learn task representations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Compared to strong baselines, our simple methods perform well in natural language inference (MNLI) and the cross-domain sentiment classification task. We even outperform unsupervised domain adaptation methods such as DANN and DSN in sentiment classification, and we are within 0.85 fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter

READ FULL TEXT

page 2

page 7

10/22/2020

Knowledge Distillation for BERT Unsupervised Domain Adaptation

A pre-trained language model, BERT, has brought significant performance ...
05/04/2023

ReMask: A Robust Information-Masking Approach for Domain Counterfactual Generation

Domain shift is a big challenge in NLP, thus, many approaches resort to ...
12/28/2020

Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy

Reducing feature redundancy has shown beneficial effects for improving t...
08/15/2022

Unsupervised Video Domain Adaptation: A Disentanglement Perspective

Unsupervised video domain adaptation is a practical yet challenging task...
05/26/2022

Unsupervised Reinforcement Adaptation for Class-Imbalanced Text Classification

Class imbalance naturally exists when train and test models in different...
02/14/2022

Domain Adaptation via Prompt Learning

Unsupervised domain adaption (UDA) aims to adapt models learned from a w...
03/30/2020

Gradually Vanishing Bridge for Adversarial Domain Adaptation

In unsupervised domain adaptation, rich domain-specific characteristics ...

Please sign up or login with your details

Forgot password? Click here to reset