OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

07/05/2023
βˆ™
by   Shengding Hu, et al.
βˆ™
0
βˆ™

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.

READ FULL TEXT

page 3

page 5

research
βˆ™ 02/20/2022

𝒴-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning

With the success of large-scale pre-trained models (PTMs), how efficient...
research
βˆ™ 04/26/2023

PVP: Pre-trained Visual Parameter-Efficient Tuning

Large-scale pre-trained transformers have demonstrated remarkable succes...
research
βˆ™ 09/30/2022

Prompt Tuning for Graph Neural Networks

In recent years, prompt tuning has set off a research boom in the adapta...
research
βˆ™ 05/23/2022

When does Parameter-Efficient Transfer Learning Work for Machine Translation?

Parameter-efficient fine-tuning methods (PEFTs) offer the promise of ada...
research
βˆ™ 07/07/2021

ADAPT : Awesome Domain Adaptation Python Toolbox

ADAPT is an open-source python library providing the implementation of s...
research
βˆ™ 04/30/2023

Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation

Recently, transformers have shown strong ability as visual feature extra...
research
βˆ™ 05/30/2023

Prompt-based Tuning of Transformer Models for Multi-Center Medical Image Segmentation

Medical image segmentation is a vital healthcare endeavor requiring prec...

Please sign up or login with your details

Forgot password? Click here to reset