Unsupervised Summarization Re-ranking - A Study on Candidate Re-ranking for Unsupervised Abstract Summaries

Aug 21, 2026

Upload by Eden

Unsupervised abstractive summarization models such as PEGASUS deliver appealing zero-shot results, yet their output quality varies widely across generated candidates and remains far behind supervised approaches. This paper introduces an unsupervised re-ranking strategy that selects the best summary candidate without additional labeled training, aiming to reduce the performance gap. Experiments show improved ROUGE relative to the unsupervised PEGASUS baseline and ChatGPT across multiple benchmarks, with gains averaged over many zero-shot transfer setups.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

AccessFree to download

Abstract

Introduction

Unsupervised Summarization Re-ranking Mathieu Ravaut 1 , 2 , Shafiq Joty∗1 , 3 Nancy F. Chen2 1 Nanyang Technological University, Singapore 2 Institute of Infocomm Research (I2R), A∗ STAR, Singapore 3 Salesforce AI {mathieuj001@e.ntu, [srjoty@ntu}.edu.sg](srjoty@ntu}.edu.sg)[ ](srjoty@ntu}.edu.sg)[nfychen@i2r.a-star.edu.sg](nfychen@i2r.a-star.edu.sg) Abstract With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51%(up to 23.73% from XSum to WikiHow) averagedover 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another) .1 1 Introduction Transformer-based encoder-decoder language models have achieved great success in abstractive summarization in the last few years, and produce fluent summaries which can be quite abstractive (Raffelet al., 2019 ; Lewis et al., 2020 ; Zhang et al., 2020) . These models follow the pre-train then fine-tune paradigm: they are first pre-trained with a selfsupervised objective on a large text corpus; then they are fine-tuned on the downstream dataset of interest, using the available supervision, which maybe very scarce. Finding a better pre-training objective remains an active research area. Some models like T5 (Raffel et al., 2019) and BART (Lewis et al., 2020) adopt a more general language modeling objective (e.g., masked span generation), while * Work done when the author was on leave from NTU. 1Code for all experiments are available at [https://](https://)[ ](https://)[github.com/ntunlp/SummScore](github.com/ntunlp/SummScore). | Generation method | Summary candidate | R-1 | R-2 | R-L | | --- | --- | --- | --- | --- | | Beam search | First (top beam) | 35.47 | 13.89 | 31.61 | | | Random | 34.89 | 13.46 | 31.22 | | | Minimum | 26.64 | 7.68 | 23.18 | | | Maximum (oracle) | 42.62 | 19.76 | 38.75 | | Diverse beam search | First | 34.35 | 13.02 | 30.65 | | | Random | 31.73 | 11.22 | 28.4 | | | Minimum | 21.25 | 4.45 | 18.61 | | | Maximum (oracle) | 41.87 | 19.29 | 38.22 | | Nucleus sampling | First | 32.14 | 11.29 | 28.66 | | | Random | 32.12 | 11.29 | 28.64 | | | Minimum | 24.09 | 6.49 | 21.19 | | | Maximum (oracle) | 40.19 | 17.47 | 36.43 | Table 1: ROUGE results with PEGASUS (unsupervised) on CNN/DM test set, for three generation methods to produce 20 summary candidates, and four candidate selection strategies. R-1, R-2, R-L stands for ROUGE-1/2/L. others like PEGASUS (Zhang et al., 2020) or TED (Yang et al., 2020) are pre-trained specifically for the task of summarizing a document. PEGASUS uses salient sentences of the document as a proxy summary label, while TED leverages the lead bias to get the pseudo-summary target. Despite the impressive success on supervised abstractive summarization tasks, unsupervised summarization remains very challenging. The LEAD-3 (extractive) baseline which simply takes the first three sentences of a document as its summary, remains far ahead of unsupervised approaches on several news summarization datasets (See et al., 2017), especially the popular CNN/DM dataset (Hermann et al., 2015) . In fact, it was only improved on by supervised abstractive models not more than five years ago (Narayan et al., 2018) . It is expected that a model which has never

FAQs

Why do unsupervised summarization models need re-ranking?

Unsupervised models generate many summary candidates, but quality varies greatly while only one candidate is retained. Re-ranking helps pick better candidates without relying on supervision.

What does the proposed approach optimize in the unsupervised setting?

The method re-ranks summary candidates using an unsupervised procedure, with the goal of narrowing the gap between unsupervised and supervised summarization performance.

How is effectiveness measured in the experiments?

Performance is evaluated using ROUGE metrics (ROUGE-1/2/L) across several widely used summarization benchmarks and many zero-shot transfer setups.

Aug 21, 2026

Upload by Eden