Do Multi-Document Summarization Models Synthesize?

by   Jay DeYoung, et al.

Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to a key property or aspect. For example, a synopsis of film reviews all written about a particular movie should reflect the average critic consensus. As a more consequential example, consider narrative summaries that accompany biomedical systematic reviews of clinical trial results. These narratives should fairly summarize the potentially conflicting results from individual trials. In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this type of synthesis? To assess this we perform a suite of experiments that probe the degree to which conditional generation models trained for summarization using standard methods yield outputs that appropriately synthesize inputs. We find that existing models do partially perform synthesis, but do so imperfectly. In particular, they are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., the ratio of positive to negative movie reviews). We propose a simple, general method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate. This approach improves model synthesis performance. We hope highlighting the need for synthesis (in some summarization settings), motivates further research into multi-document summarization methods and learning objectives that explicitly account for the need to synthesize.


page 1

page 2

page 3

page 4


Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

We consider the problem of automatically generating a narrative biomedic...

Unsupervised Neural Multi-document Abstractive Summarization

Abstractive summarization has been studied using neural sequence transdu...

DynE: Dynamic Ensemble Decoding for Multi-Document Summarization

Sequence-to-sequence (s2s) models are the basis for extensive work in na...

Massive Multi-Document Summarization of Product Reviews with Weak Supervision

Product reviews summarization is a type of Multi-Document Summarization ...

MS2: Multi-Document Summarization of Medical Studies

To assess the effectiveness of any medical intervention, researchers mus...

Content based Weighted Consensus Summarization

Multi-document summarization has received a great deal of attention in t...

Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges

We present TrialsSummarizer, a system that aims to automatically summari...

Please sign up or login with your details

Forgot password? Click here to reset