Distilling Causal Effect of Data in Class-Incremental Learning

by   Xinting Hu, et al.

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the framework, we find that although the feature/label distillation is storage-efficient, its causal effect is not coherent with the end-to-end feature learning merit, which is however preserved by data replay. To this end, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72


page 1

page 2

page 3

page 4


Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

This paper proposes two novel knowledge transfer techniques for class-in...

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition

Continual Learning for Named Entity Recognition (CL-NER) aims to learn a...

Condensed Prototype Replay for Class Incremental Learning

Incremental learning (IL) suffers from catastrophic forgetting of old ta...

Causal Incremental Graph Convolution for Recommender System Retraining

Real-world recommender system needs to be regularly retrained to keep wi...

Online Deep Metric Learning via Mutual Distillation

Deep metric learning aims to transform input data into an embedding spac...

Incremental Learning with Unlabeled Data in the Wild

Deep neural networks are known to suffer from catastrophic forgetting in...

Rotation Augmented Distillation for Exemplar-Free Class Incremental Learning with Detailed Analysis

Class incremental learning (CIL) aims to recognize both the old and new ...

Please sign up or login with your details

Forgot password? Click here to reset