TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation

by   Jinyu Yang, et al.

Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge, however, remains unexplored in the literature. To fill this gap, this paper first comprehensively investigates the transferability of ViT on a variety of domain adaptation tasks. Surprisingly, ViT demonstrates superior transferability over its CNNs-based counterparts with a large margin, while the performance can be further improved by incorporating adversarial adaptation. Notwithstanding, directly using CNNs-based adaptation strategies fails to take the advantage of ViT's intrinsic merits (e.g., attention mechanism and sequential image representation) which play an important role in knowledge transfer. To remedy this, we propose an unified framework, namely Transferable Vision Transformer (TVT), to fully exploit the transferability of ViT for domain adaptation. Specifically, we delicately devise a novel and effective unit, which we term Transferability Adaption Module (TAM). By injecting learned transferabilities into attention blocks, TAM compels ViT focus on both transferable and discriminative features. Besides, we leverage discriminative clustering to enhance feature diversity and separation which are undermined during adversarial domain alignment. To verify its versatility, we perform extensive studies of TVT on four benchmarks and the experimental results demonstrate that TVT attains significant improvements compared to existing state-of-the-art UDA methods.


Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a l...

Domain Adaptation via Bidirectional Cross-Attention Transformer

Domain Adaptation (DA) aims to leverage the knowledge learned from a sou...

Stochastic Adversarial Gradient Embedding for Active Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims to bridge the gap between a so...

Measuring the Feasibility of Analogical Transfer using Complexity

Analogies are 4-ary relations of the form "A is to B as C is to D". Whil...

Safe Self-Refinement for Transformer-based Domain Adaptation

Unsupervised Domain Adaptation (UDA) aims to leverage a label-rich sourc...

Universal Domain Adaptation via Compressive Attention Matching

Universal domain adaptation (UniDA) aims to transfer knowledge from the ...

UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation

Aiming towards human-level generalization, there is a need to explore ad...

Please sign up or login with your details

Forgot password? Click here to reset