𝒴-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning

02/20/2022
βˆ™
by   Yitao Liu, et al.
βˆ™
0
βˆ™

With the success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Although some parameter-efficient tuning paradigms have been proposed to address this problem, they still require large resources to compute the gradients in the training phase. In this paper, we propose 𝒴-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. 𝒴-tuning learns dense representations for labels 𝒴 defined in a given task and aligns them to fixed feature representation. Without tuning the features of input text and model parameters, 𝒴-tuning is both parameter-efficient and training-efficient. For DeBERTa_XXL with 1.6 billion parameters, 𝒴-tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
βˆ™ 07/05/2023

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

The scale of large pre-trained models (PTMs) poses significant challenge...
research
βˆ™ 05/17/2023

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

It has become a popular paradigm to transfer the knowledge of large-scal...
research
βˆ™ 03/31/2022

Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models

Prompting has recently become a popular paradigm for adapting language m...
research
βˆ™ 06/14/2021

Pre-Trained Models: Past, Present and Future

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently...
research
βˆ™ 04/12/2023

Global Prompt Cell: A Portable Control Module for Effective Prompt

As a novel approach to tuning pre-trained models, prompt tuning involves...
research
βˆ™ 07/21/2023

Tuning Pre-trained Model via Moment Probing

Recently, efficient fine-tuning of large-scale pre-trained models has at...
research
βˆ™ 04/12/2020

Gradients as Features for Deep Representation Learning

We address the challenging problem of deep representation learning–the e...

Please sign up or login with your details

Forgot password? Click here to reset