𝒴-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning

by   Yitao Liu, et al.

With the success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Although some parameter-efficient tuning paradigms have been proposed to address this problem, they still require large resources to compute the gradients in the training phase. In this paper, we propose 𝒴-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. 𝒴-tuning learns dense representations for labels 𝒴 defined in a given task and aligns them to fixed feature representation. Without tuning the features of input text and model parameters, 𝒴-tuning is both parameter-efficient and training-efficient. For DeBERTa_XXL with 1.6 billion parameters, 𝒴-tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.


page 1

page 2

page 3

page 4

βˆ™ 07/05/2023

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

The scale of large pre-trained models (PTMs) poses significant challenge...
βˆ™ 05/17/2023

G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks

It has become a popular paradigm to transfer the knowledge of large-scal...
βˆ™ 03/31/2022

Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models

Prompting has recently become a popular paradigm for adapting language m...
βˆ™ 06/14/2021

Pre-Trained Models: Past, Present and Future

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently...
βˆ™ 04/12/2023

Global Prompt Cell: A Portable Control Module for Effective Prompt

As a novel approach to tuning pre-trained models, prompt tuning involves...
βˆ™ 07/21/2023

Tuning Pre-trained Model via Moment Probing

Recently, efficient fine-tuning of large-scale pre-trained models has at...
βˆ™ 04/12/2020

Gradients as Features for Deep Representation Learning

We address the challenging problem of deep representation learning–the e...

Please sign up or login with your details

Forgot password? Click here to reset