Attention-based Class Activation Diffusion for Weakly-Supervised Semantic Segmentation

by   Jianqiang Huang, et al.

Extracting class activation maps (CAM) is a key step for weakly-supervised semantic segmentation (WSSS). The CAM of convolution neural networks fails to capture long-range feature dependency on the image and result in the coverage on only foreground object parts, i.e., a lot of false negatives. An intuitive solution is “coupling” the CAM with the long-range attention matrix of visual transformers (ViT) We find that the direct “coupling”, e.g., pixel-wise multiplication of attention and activation, achieves a more global coverage (on the foreground), but unfortunately goes with a great increase of false positives, i.e., background pixels are mistakenly included. This paper aims to tackle this issue. It proposes a new method to couple CAM and Attention matrix in a probabilistic Diffusion way, and dub it AD-CAM. Intuitively, it integrates ViT attention and CAM activation in a conservative and convincing way. Conservative is achieved by refining the attention between a pair of pixels based on their respective attentions to common neighbors, where the intuition is two pixels having very different neighborhoods are rarely dependent, i.e., their attention should be reduced. Convincing is achieved by diffusing a pixel's activation to its neighbors (on the CAM) in proportion to the corresponding attentions (on the AM). In experiments, our results on two challenging WSSS benchmarks PASCAL VOC and MS COCO show that AD-CAM as pseudo labels can yield stronger WSSS models than the state-of-the-art variants of CAM.


page 2

page 13

page 14


Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds

Weakly-supervised semantic segmentation (WSSS) has recently gained much ...

TransCAM: Transformer Attention-based CAM Refinement for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) with only image-level sup...

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization

Weakly supervised object localization (WSOL) is a challenging problem wh...

Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation

Extracting class activation maps (CAM) is arguably the most standard ste...

All-pairs Consistency Learning for Weakly Supervised Semantic Segmentation

In this work, we propose a new transformer-based regularization to bette...

Extracting Class Activation Maps from Non-Discriminative Features as well

Extracting class activation maps (CAM) from a classification model often...

Mitigating Biased Activation in Weakly-supervised Object Localization via Counterfactual Learning

In this paper, we focus on an under-explored issue of biased activation ...

Please sign up or login with your details

Forgot password? Click here to reset