Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation

03/07/2023

∙

by   Yixin Liu, et al.

∙

Carnegie Mellon University

∙

Yale University

∙

Salesforce

∙

13

∙

share

Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. In this work, we develop strong-performing automatic metrics for reference-based summarization evaluation, based on a two-stage evaluation pipeline that first extracts basic information units from one text sequence and then checks the extracted units in another sequence. The metrics we developed include two-stage metrics that can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interoperability. We make the developed tools publicly available through a Python package and GitHub.