Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

by   Longquan Jiang, et al.

Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at as well as


page 1

page 2

page 3

page 4


LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs

The medical conversational system can relieve the burden of doctors and ...

BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering

Unsupervised commonsense reasoning (UCR) is becoming increasingly popula...

Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer

Knowledge graphs (KG) are essential background knowledge providers in ma...

SAGC-A68: a space access graph dataset for the classification of spaces and space elements in apartment buildings

The analysis of building models for usable area, building safety, and en...

Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases

Existing studies on question answering on knowledge bases (KBQA) mainly ...

PyGraft: Configurable Generation of Schemas and Knowledge Graphs at Your Fingertips

Knowledge graphs (KGs) have emerged as a prominent data representation a...

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

The ability to have the same experience for different user groups (i.e.,...

Please sign up or login with your details

Forgot password? Click here to reset