Enhancing Recommendation Diversity by Re-ranking with Large Language Models - arXiv 2401.11506v2

Aug 23, 2026

Upload by Kyle

The paper addresses the limitation of recommender systems that optimize only for relevance, arguing that users require meaningful choice through diversity in recommendation sets. It investigates how Large Language Models can be integrated into the recommender pipeline specifically for diversity re-ranking. An initial informal study is followed by a rigorous zero-shot prompt-based methodology producing diverse rankings. Extensive experiments compare GPT and Llama models against random and traditional re-rankers, showing better trade-offs than random baselines while still lagging behind classical methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages39

Reading Time1 h 38 min

AccessFree to download

Introduction

Motivation: relevance is not enough

LLMs for recommendation pipelines

Methodology

Informal verification of diversity understanding

Zero-shot prompt templates for diverse re-ranking

Experiments

Tested LLM families and baselines

Comparison to random and traditional re-rankers

Reproducibility via open-source code

Findings

Performance and cost trade-offs

Comparison to traditional methods

Outlook on competitiveness

arXiv :2401 . 1 1506v2 [ cs .IR] 17 Jun 2024 Enhancing Recommendation Diversity by Re-ranking with Large Language Models DIEGO CARRARO, Insight Centre for Data Analytics, School of Computer Science & IT, University College Cork, Ireland DEREK BRIDGE, Insight Centre for Data Analytics, School of Computer Science & IT, University College Cork, Ireland It has long been recognized that it is not enough for a Recommender System (RS) to provide recommendations based only on their relevance to users. Among many other criteria, the set of recommendations may need to be diverse. Diversity is one way of handling recommendation uncertainty and ensuring that recommendations oﬀer users a meaningful choice. The literature reports many ways of measuring diversity and improving the diversity of a set of recommendations, most notably by re-ranking and selecting from a larger set of candidate recommendations. Driven by promising insights from the literature on how to incorporate versatile Large Language Models (LLMs) into the RS pipeline, in this paper we show how LLMs can be used for diversity re-ranking. We begin with an informal study that veriﬁes that LLMs can be used for re-ranking tasks and do have some understanding of the concept of item diversity. Then, we design a more rigorous methodology where LLMs are prompted to generate a diverse ranking from a candidate ranking using various prompt templates with diﬀerent re-ranking instructions in a zero-shot fashion. We conduct comprehensive experiments testing state-of-the-art LLMs from the GPT and Llama families. We compare their re-ranking capabilities with random re-ranking and various traditional re-ranking methods from the literature. We open-source the code of our experiments for reproducibility. Our ﬁndings suggest that the trade-oﬀs (in terms of performance and costs, among others) of LLM-based rerankers are superior to those of random re-rankers but, as yet, inferior to the ones of traditional re-rankers. However, the LLM approach is promising. LLMs exhibit improved performance on many natural language processing and recommendation tasks and lower inference costs. Given these trends, we can expect LLM-based re-ranking to become more competitive soon. CCS Concepts: • Information systems → Information retrieval diversity; Language models; Recommender systems. Additional Key Words and Phrases: Recommender Systems, Large Language Models, Diversity, Re-ranking 1 INTRODUCTION Large Language Models (LLMs) have rapidly become a breakthrough technology since the advent of their popular pioneers GPT [49] and BERT [13], introduced in 2018 . Since then, many more LLMs with enhanced capabilities have been proposed, such as ChatGPT, Bard and Llama. They can perform various language-related tasks, including, for example, translation, summarization and conversation; and they give the appearance of understanding complex contexts and exhibiting reasoning, planning and problem-solving capabilities [35, 58, 70] . LLMs are typically pre-trained with large datasets of text to serve as general-purpose models and then adapted to diﬀerent downstream tasks and domains by a Authors’ addresses: Diego Carraro, Insight Centre for Data Analytics, School of Computer Science& IT, University College Cork, Ireland, diego.carraro@ [insight-centre.org](insight-centre.org); Derek Bridge, Insight Centre for Data Analytics, School of Computer Science & IT, University College Cork, Ireland, derek.bridge@ [insight-centre.org](insight-centre.org). Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or commercial advantage and that copies bear this notice and the full citation on the ﬁrst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s) . © 2024 Copyright held by the owner/author(s) . Manuscript submit

FAQs

Why does the paper focus on recommendation diversity instead of only relevance?

Recommender systems that optimize only relevance may not provide users with a meaningful choice. The paper argues that diverse recommendation sets address uncertainty and better satisfy user needs.

How do the authors use large language models for diversity re-ranking?

They prompt LLMs in a zero-shot manner to generate a diverse ranking from a candidate ranking. Multiple prompt templates implement different diversity re-ranking instructions.

What comparisons are made in the experiments and what are the main results?

Experiments test state-of-the-art LLMs from GPT and Llama families and compare them with random re-ranking and traditional re-ranking methods. LLM-based re-rankers outperform random baselines in trade-offs, though they remain inferior to traditional re-rankers so far, while still being promising.

Aug 23, 2026

Upload by Kyle