Mirror Complementary Transformer Network for RGB-thermal Salient Object Detection

by   Xiurong Jiang, et al.

RGB-thermal salient object detection (RGB-T SOD) aims to locate the common prominent objects of an aligned visible and thermal infrared image pair and accurately segment all the pixels belonging to those objects. It is promising in challenging scenes such as nighttime and complex backgrounds due to the insensitivity to lighting conditions of thermal images. Thus, the key problem of RGB-T SOD is to make the features from the two modalities complement and adjust each other flexibly, since it is inevitable that any modalities of RGB-T image pairs failure due to challenging scenes such as extreme light conditions and thermal crossover. In this paper, we propose a novel mirror complementary Transformer network (MCNet) for RGB-T SOD. Specifically, we introduce a Transformer-based feature extraction module to effective extract hierarchical features of RGB and thermal images. Then, through the attention-based feature interaction and serial multiscale dilated convolution (SDC) based feature fusion modules, the proposed model achieves the complementary interaction of low-level features and the semantic fusion of deep features. Finally, based on the mirror complementary structure, the salient regions of the two modalities can be accurately extracted even one modality is invalid. To demonstrate the robustness of the proposed model under challenging scenes in real world, we build a novel RGB-T SOD dataset VT723 based on a large public semantic segmentation RGB-T dataset used in the autonomous driving domain. Expensive experiments on benchmark and VT723 datasets show that the proposed method outperforms state-of-the-art approaches, including CNN-based and Transformer-based methods. The code and dataset will be released later at https://github.com/jxr326/SwinMCNet.


page 1

page 4

page 5

page 6

page 7

page 9

page 10

page 12


Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks

Recently, RGB-Thermal based perception has shown significant advances. T...

RGBT Salient Object Detection: A Large-scale Dataset and Benchmark

Salient object detection in complex scenes and environments is a challen...

Glass Segmentation with RGB-Thermal Image Pairs

This paper proposes a new glass segmentation method utilizing paired RGB...

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

In recent years, RGB-T salient object detection (SOD) has attracted cont...

Scribble-Supervised RGB-T Salient Object Detection

Salient object detection segments attractive objects in scenes. RGB and ...

Position-Aware Relation Learning for RGB-Thermal Salient Object Detection

RGB-Thermal salient object detection (SOD) combines two spectra to segme...

Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection

RGB-Thermal (RGB-T) pedestrian detection aims to locate the pedestrians ...

Please sign up or login with your details

Forgot password? Click here to reset