Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving

by   Xiwen Liang, et al.

Aiming towards a holistic understanding of multiple downstream tasks simultaneously, there is a need for extracting features with better transferability. Though many latest self-supervised pre-training methods have achieved impressive performance on various vision tasks under the prevailing pretrain-finetune paradigm, their generalization capacity to multi-task learning scenarios is yet to be explored. In this paper, we extensively investigate the transfer performance of various types of self-supervised methods, e.g., MoCo and SimCLR, on three downstream tasks, including semantic segmentation, drivable area segmentation, and traffic object detection, on the large-scale driving dataset BDD100K. We surprisingly find that their performances are sub-optimal or even lag far behind the single-task baseline, which may be due to the distinctions of training objectives and architectural design lied in the pretrain-finetune paradigm. To overcome this dilemma as well as avoid redesigning the resource-intensive pre-training stage, we propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training, where the off-the-shelf pretrained models can be effectively adapted without increasing the training overhead. During the adapt stage, we utilize learnable multi-scale adapters to dynamically adjust the pretrained model weights supervised by multi-task objectives while leaving the pretrained knowledge untouched. Furthermore, we regard the vision-language pre-training model CLIP as a strong complement to the pretrain-adapt-finetune paradigm and propose a novel adapter named LV-Adapter, which incorporates language priors in the multi-task model via task-specific prompting and alignment between visual and textual features.


page 2

page 6


Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving

Multi-task learning has emerged as a powerful paradigm to solve a range ...

GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learning

Pre-training over mixtured multi-task, multi-domain, and multi-modal dat...

Region Proposal Network Pre-Training Helps Label-Efficient Object Detection

Self-supervised pre-training, based on the pretext task of instance disc...

Pre-train, Adapt and Detect: Multi-Task Adapter Tuning for Camouflaged Object Detection

Camouflaged object detection (COD), aiming to segment camouflaged object...

An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training

We present a model that can perform multiple vision tasks and can be ada...

Curriculum Modeling the Dependence among Targets with Multi-task Learning for Financial Marketing

Multi-task learning for various real-world applications usually involves...

Visual Prompting via Image Inpainting

How does one adapt a pre-trained visual model to novel downstream tasks ...

Please sign up or login with your details

Forgot password? Click here to reset