Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping

04/05/2023
by   Burak Ekim, et al.
0

Jointly harnessing complementary features of multi-modal input data in a common latent space has been found to be beneficial long ago. However, the influence of each modality on the models decision remains a puzzle. This study proposes a deep learning framework for the modality-level interpretation of multimodal earth observation data in an end-to-end fashion. While leveraging an explainable machine learning method, namely Occlusion Sensitivity, the proposed framework investigates the influence of modalities under an early-fusion scenario in which the modalities are fused before the learning process. We show that the task of wilderness mapping largely benefits from auxiliary data such as land cover and night time light data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/06/2021

Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks

Multi-modality data is becoming readily available in remote sensing (RS)...
research
12/28/2022

Multimodal Explainability via Latent Shift applied to COVID-19 stratification

We are witnessing a widespread adoption of artificial intelligence in he...
research
08/17/2023

Predicting Crop Yield With Machine Learning: An Extensive Analysis Of Input Modalities And Models On a Field and sub-field Level

We introduce a simple yet effective early fusion method for crop yield p...
research
11/14/2022

PMR: Prototypical Modal Rebalance for Multimodal Learning

Multimodal learning (MML) aims to jointly exploit the common priors of d...
research
04/08/2021

Multimodal Fusion Refiner Networks

Tasks that rely on multi-modal information typically include a fusion mo...
research
12/12/2020

Multimodal In-bed Pose and Shape Estimation under the Blankets

Humans spend vast hours in bed – about one-third of the lifetime on aver...
research
01/12/2023

Multimodal Deep Learning

This book is the result of a seminar in which we reviewed multimodal app...

Please sign up or login with your details

Forgot password? Click here to reset