ReviewQA: a relational aspect-based opinion reading dataset

by   Quentin Grail, et al.

Deep reading models for question-answering have demonstrated promising performance over the last couple of years. However current systems tend to learn how to cleverly extract a span of the source document, based on its similarity with the question, instead of seeking for the appropriate answer. Indeed, a reading machine should be able to detect relevant passages in a document regarding a question, but more importantly, it should be able to reason over the important pieces of the document in order to produce an answer when it is required. To motivate this purpose, we present ReviewQA, a question-answering dataset based on hotel reviews. The questions of this dataset are linked to a set of relational understanding competencies that we expect a model to master. Indeed, each question comes with an associated type that characterizes the required competency. With this framework, it is possible to benchmark the main families of models and to get an overview of what are the strengths and the weaknesses of a given model on the set of tasks evaluated in this dataset. Our corpus contains more than 500.000 questions in natural language over 100.000 hotel reviews. Our setup is projective, the answer of a question does not need to be extracted from a document, like in most of the recent datasets, but selected among a set of candidates that contains all the possible answers to the questions of the dataset. Finally, we present several baselines over this dataset.


page 1

page 2

page 3

page 4


Know What You Don't Know: Unanswerable Questions for SQuAD

Extractive reading comprehension systems can often locate the correct an...

CodeQA: A Question Answering Dataset for Source Code Comprehension

We propose CodeQA, a free-form question answering dataset for the purpos...

AmazonQA: A Review-Based Question Answering Task

Every day, thousands of customers post questions on Amazon product pages...

MeeQA: Natural Questions in Meeting Transcripts

We present MeeQA, a dataset for natural-language question answering over...

RikiNet: Reading Wikipedia Pages for Natural Question Answering

Reading long documents to answer open-domain questions remains challengi...

Iterative Multi-document Neural Attention for Multiple Answer Prediction

People have information needs of varying complexity, which can be solved...

Meta Answering for Machine Reading

We investigate a framework for machine reading, inspired by real world i...

Please sign up or login with your details

Forgot password? Click here to reset