DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation

by   Jingfan Chen, et al.

The Class Incremental Semantic Segmentation (CISS) extends the traditional segmentation task by incrementally learning newly added classes. Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN, to address the issues of catastrophic forgetting and privacy concerns. However, the generated images lack semantic precision and exhibit out-of-distribution characteristics, resulting in inaccurate masks that further degrade the segmentation performance. To tackle these challenges, we propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions (e.g., text prompts or edge maps). Specifically, DiffusePast introduces a dual-generator paradigm, which focuses on generating old class images that align with the distribution of downstream datasets while preserving the structure and layout of the original images, enabling more precise masks. To adapt to the novel visual concepts of newly added classes continuously, we incorporate class-wise token embedding when updating the dual-generator. Moreover, we assign adequate pseudo-labels of old classes to the background pixels in the new step images, further mitigating the forgetting of previously learned knowledge. Through comprehensive experiments, our method demonstrates competitive performance across mainstream benchmarks, striking a better balance between the performance of old and novel classes.


page 7

page 8


Gradient-Semantic Compensation for Incremental Semantic Segmentation

Incremental semantic segmentation aims to continually learn the segmenta...

RECALL: Replay-based Continual Learning in Semantic Segmentation

Deep networks allow to obtain outstanding results in semantic segmentati...

Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

This paper proposes two novel knowledge transfer techniques for class-in...

Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay

Endoscopy is a widely used technique for the early detection of diseases...

Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning

Class incremental learning (CIL) aims to incrementally update a trained ...

Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation

Incremental or continual learning has been extensively studied for image...

Evidential Deep Learning for Class-Incremental Semantic Segmentation

Class-Incremental Learning is a challenging problem in machine learning ...

Please sign up or login with your details

Forgot password? Click here to reset