A Comparative Study on Structural and Semantic Properties of Sentence Embeddings

by   Alexander Kalinowski, et al.

Sentence embeddings encode natural language sentences as low-dimensional dense vectors. A great deal of effort has been put into using sentence embeddings to improve several important natural language processing tasks. Relation extraction is such an NLP task that aims at identifying structured relations defined in a knowledge base from unstructured text. A promising and more efficient approach would be to embed both the text and structured knowledge in low-dimensional spaces and discover semantic alignments or mappings between them. Although a number of techniques have been proposed in the literature for embedding both sentences and knowledge graphs, little is known about the structural and semantic properties of these embedding spaces in terms of relation extraction. In this paper, we investigate the aforementioned properties by evaluating the extent to which sentences carrying similar senses are embedded in close proximity sub-spaces, and if we can exploit that structure to align sentences to a knowledge graph. We propose a set of experiments using a widely-used large-scale data set for relation extraction and focusing on a set of key sentence embedding methods. We additionally provide the code for reproducing these experiments at https://github.com/akalino/semantic-structural-sentences. These embedding methods cover a wide variety of techniques ranging from simple word embedding combination to transformer-based BERT-style model. Our experimental results show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.


page 1

page 8


SEE: Syntax-aware Entity Embedding for Neural Relation Extraction

Distant supervised relation extraction is an efficient approach to scale...

Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences

Sentence embedding methods offer a powerful approach for working with sh...

Exploring the Suitability of Semantic Spaces as Word Association Models for the Extraction of Semantic Relationships

Given the recent advances and progress in Natural Language Processing (N...

Modeling Multi-Granularity Hierarchical Features for Relation Extraction

Relation extraction is a key task in Natural Language Processing (NLP), ...

R^2-Net: Relation of Relation Learning Network for Sentence Semantic Matching

Sentence semantic matching is one of the fundamental tasks in natural la...

Logic Constrained Pointer Networks for Interpretable Textual Similarity

Systematically discovering semantic relationships in text is an importan...

Dependency-based Text Graphs for Keyphrase and Summary Extraction with Applications to Interactive Content Retrieval

We build a bridge between neural network-based machine learning and grap...

Please sign up or login with your details

Forgot password? Click here to reset