Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability

by   Alberto Santamaria-Pang, et al.

Recent advances in methods focused on the grounding problem have resulted in techniques that can be used to construct a symbolic language associated with a specific domain. Inspired by how humans communicate complex ideas through language, we developed a generalized Symbolic Semantic (S^2) framework for interpretable segmentation. Unlike adversarial models (e.g., GANs), we explicitly model cooperation between two agents, a Sender and a Receiver, that must cooperate to achieve a common goal. The Sender receives information from a high layer of a segmentation network and generates a symbolic sentence derived from a categorical distribution. The Receiver obtains the symbolic sentences and co-generates the segmentation mask. In order for the model to converge, the Sender and Receiver must learn to communicate using a private language. We apply our architecture to segment tumors in the TCGA dataset. A UNet-like architecture is used to generate input to the Sender network which produces a symbolic sentence, and a Receiver network co-generates the segmentation mask based on the sentence. Our Segmentation framework achieved similar or better performance compared with state-of-the-art segmentation methods. In addition, our results suggest direct interpretation of the symbolic sentences to discriminate between normal and tumor tissue, tumor morphology, and other image characteristics.


page 5

page 7


Symbolic Semantic Segmentation and Interpretation of COVID-19 Lung Infections in Chest CT volumes based on Emergent Languages

The coronavirus disease (COVID-19) has resulted in a pandemic crippling ...

Reasoning and Algorithm Selection Augmented Symbolic Segmentation

In this paper we present an alternative method to symbolic segmentation:...

ESCELL: Emergent Symbolic Cellular Language

We present ESCELL, a method for developing an emergent symbolic language...

Symbolic Segmentation Using Algorithm Selection

In this paper we present an alternative approach to symbolic segmentatio...

SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark

Existing work in language grounding typically study single environments....

Neuro-Symbolic Causal Reasoning Meets Signaling Game for Emergent Semantic Communications

Semantic communication (SC) aims to communicate reliably with minimal da...

Please sign up or login with your details

Forgot password? Click here to reset