RouteRec: 严格评估推荐代理的选择与聚合

Jul 16, 2026

Upload by Maeve

Recommender systems face a heterogeneous-agents choice where collaborative filters, sequential models, content retrievers, and LLM rerankers each have different costs and strengths. The study introduces RouteRec, a task-aware agent-ranking framework under cost constraints, comparing request-level hard selection versus item-level learned aggregation across four traditional agents and one LLM reranker. On MovieLens-1M, the oracle shows strong headroom (HR@10=0.584), yet leakage-free 5-fold evaluation shows hard selection underperforms BM25, while item-level learned aggregation improves quality and uses LLM calls selectively.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction

Motivation for request-level routing

RouteRec framework and decision stages

Related Work

LLMs for recommendation

Hybrid recommenders and algorithm selection

RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation Kaiji Zhou University of Birmingham Birmingham, United Kingdom [kxz571@student.bham.ac.uk](kxz571@student.bham.ac.uk) Vladimir Kalmykov University of Birmingham Birmingham, United Kingdom [vxk276@student.bham.ac.uk](vxk276@student.bham.ac.uk) Yue Feng∗ University of Birmingham Birmingham, United Kingdom [y.feng.6@bham.ac.uk](y.feng.6@bham.ac.uk) arXiv :2607 .09908v 1 [ cs .CL] 10 Jul 2026 Abstract Recommender systems increasingly face a choice among heterogeneous agents—collaborative filters, sequential models, contentbased retrievers, and LLM-based rerankers—yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent. On MovieLens-1M, the full quality oracle has substantial headroom (HR@10=0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0 . 223 vs. 0. 254), and selective LLM escalation does not improve it. The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10= 0.295 with 70.2% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space. 1 Introduction Recent work has explored LLMs as recommendation agents—as zero-shot rankers [9], instruction followers [30], and tool users [31] . However, LLM inference can be substantially more expensive than conventional recommendation inference [3, 18], and LLMs do not uniformly dominate cheaper alternatives. Traditional recommendation methods remain competitive for warm users with rich interaction histories [13], while surveys discuss potential LLM advantagesin sparse, cross-domain, and explanation-oriented settings [14, 28] . This observation motivates a shift from “which single model is best?” to “which model should handle this request?”—a framing consistent with classical algorithm selection [21], meta-learning for recommendation [4, 6, 23], and LLM routing [18, 25] . We use recommender agent as an operational umbrella term for any callable recommendation service with a ranked-list interface, a cost profile, and task-dependent suitability. Accordingly, our pool contains four cheap traditional recommender agents and one expensive LLM reranker agent. This definition matches the agent-search setting, where systems must discover, compare, and invoke heterogeneous callable services; it does not require every service to be an autonomous dialogue agent. We study this routing problem through RouteRec, a lightweight framework for per-instance decisions across the five recommender agents. RouteRec-Select combines request context with cheap probe ∗ Corresponding author. Figure 1: RouteRec-Select architecture. A request is encoded and combined with cheap-probe disagreement features to select the best cheap agent (Stage 1) and decide whether to escalate to the LLM reranker (Stage 2). disagreement, a deployable measure of cheap-agent output divergence, to choose one cheap agent and decide whether to escalate to the LLM (Fig. 1) . Because hard selection returns one complete list from one agent, it can discard useful evidence from other agents. We therefore also evaluate RouteRec-Stack, which reranks the union of agent top-􀀺 lists using deployable item-level rank and score evidence. This paper makes three contributions. First, we cast search over recommender agents as task-aware utility maximisation under cost constraints and evaluate eve

FAQs

What problem does RouteRec address in recommender systems?

RouteRec targets task-aware routing among heterogeneous recommendation agents, deciding which agent(s) should handle a given request under strict cost constraints.

How does RouteRec compare request-level selection with item-level aggregation?

RouteRec-Select performs request context–based hard selection and optional LLM escalation, while RouteRec-Stack aggregates signals at the item level by reranking unions of agents’ candidate lists.

What do results on MovieLens-1M show about hard selection versus aggregation?

Under a leakage-free 5-fold out-of-fold protocol, hard selection stays below BM25, while learned aggregation—especially the gated all-agent variant—recovers more cross-agent signal and achieves higher HR@10 with selective LLM calls.

Jul 16, 2026

Upload by Maeve