ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation

by   Youngmin Oh, et al.

We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still a lack of theoretical understanding of them. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. We propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, we first show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. We also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, we introduce category-specific rotation matrices updating the features for each category separately. We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves a better trade-off in terms of accuracy and efficiency.


page 15

page 22

page 23


Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation

Modern incremental learning for semantic segmentation methods usually le...

Delving into Transformer for Incremental Semantic Segmentation

Incremental semantic segmentation(ISS) is an emerging task where old mod...

Memory Replay GANs: learning to generate images from new categories without forgetting

Previous works on sequential learning address the problem of forgetting ...

Incremental Learning Techniques for Semantic Segmentation

Deep learning architectures exhibit a critical drop of performance due t...

Continual Attentive Fusion for Incremental Learning in Semantic Segmentation

Over the past years, semantic segmentation, as many other tasks in compu...

DER: Dynamically Expandable Representation for Class Incremental Learning

We address the problem of class incremental learning, which is a core st...

Continual Semantic Segmentation with Automatic Memory Sample Selection

Continual Semantic Segmentation (CSS) extends static semantic segmentati...

Please sign up or login with your details

Forgot password? Click here to reset