Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE

Jul 16, 2026

Upload by Aurora

Generative retrieval supports large-scale recommendation, but commonly relies on supervised next-item prediction that does not directly optimize long-term user satisfaction. The work models recommendation as session-level sequential decision-making and trains autoregressive generative retrievers with off-policy REINFORCE using pre-collected data. Multi-step importance-weight approximation replaces prior one-step correction. It enables offline evaluation via a feedback model for user responses, adapting doubly robust OPE and a test-time scaling method selecting recommendations with maximum predicted long-term return.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Long-term optimization in recommender systems

Reinforcement learning for long-term rewards

On-policy vs. off-policy methods

Challenges of offline evaluation and OPE

Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE Artem Matveev [matfu21@yandex.ru](matfu21@yandex.ru)[ ](matfu21@yandex.ru)AI VK Moscow, Russia Sergei Makeev [neuralsrg@gmail.com](neuralsrg@gmail.com)[ ](neuralsrg@gmail.com)AI VK Moscow, Russia Aleksei Krasilnikov [TheKabeton@yandex.ru](TheKabeton@yandex.ru)[ ](TheKabeton@yandex.ru)AI VK Moscow, Russia Vladimir Baikalov [deadinside@itmo.ru](deadinside@itmo.ru)[ ](deadinside@itmo.ru)AI VK Moscow, Russia Sergei Liamaev [liamaev.sergei@gmail.com](liamaev.sergei@gmail.com)[ ](liamaev.sergei@gmail.com)AI VK Moscow, Russia Kirill Khrylchenko [elightelol@gmail.com](elightelol@gmail.com)[ ](elightelol@gmail.com)HSE University Moscow, Russia arXiv :2607 .028 18v 1 [ cs .IR] 2 Jul 2026 Abstract Generative retrieval has become a popular paradigm for large-scale recommendation. However, it is typically trained with supervised next-item prediction objectives that do not directly optimize longterm user satisfaction. In this work, we formulate recommendation as a session-level sequential decision-making problem and introduce an autoregressive approach for training generative retrievers with off-policy REINFORCE on pre-collected data. Unlike the one-step off-policy correction used in prior work, we propose a multi-step approximation of importance weights enabled by the autoregressive formulation. To support offline evaluation, we train a user feedback model that simulates user responses to generated recommendations. This lets us adapt doubly robust off-policy evaluation for sequential decision-making to recommendation, a setting that has received limited attention. We further introduce a feedback-modelbased test-time scaling procedure that simulates future responsesand selects recommendations with the highest predicted long-term returns. Experiments on the public large-scale Yambda-5B dataset show that our RL agent improves offline estimates of cumulative session reward over next-item and next-positive prediction baselines, while largely preserving retrieval quality. Moreover, allocating more inference-time compute to simulating future responses improves model-based long-term return estimates without updating the policy. CCS Concepts • Information systems → Information retrieval; Recommender systems; • Computing methodologies → Reinforcement learning. Keywords Off-Policy Learning, Off-Policy Evaluation, REINFORCE, Recommender Systems, Reinforcement Learning, Importance Sampling, Candidate Generation Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s) . KDD’26, Jeju Island, Republic of Korea © 2026 Copyright held by the owner/author(s) . ACM Reference Format: Artem Matveev, Sergei Makeev, Aleksei Krasilnikov, Vladimir Baikalov, Sergei Liamaev, and Kirill Khrylchenko. 2026. Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE. In Proceedings of 5th Workshop on End-End Customer Journey Optimization at KDD 2026. (KDD’26) . ACM, New York, NY, USA, 10 pages. 1 Introduction Recommender systems are widely deployed across industry to help users navigate vast collections of content. Their mission is to retain millions of users by presenting the most relevant items. To achieve this goal, recommender systems are typically optimized with respect to immediate feedback signals, such as clicks [28] or repins [31], which can be intuitively viewed as searching for a local optimum in the space of user interests. Reinforcement learning. Reinforcement learning (RL) offers a way to extend this optimization objective to account for long-term rewards

FAQs

Why does standard generative retrieval training underperform for long-term satisfaction?

It is typically trained with supervised next-item objectives, which optimize immediate prediction rather than directly maximizing long-term user satisfaction.

How does the paper formulate and train generative retrievers for long-term optimization?

It models recommendation as session-level sequential decision-making and trains autoregressive generative retrievers using off-policy REINFORCE on pre-collected data, with multi-step importance-weight approximation.

How is offline evaluation made possible in this setting?

A user feedback model simulates user responses to generated recommendations, enabling doubly robust off-policy evaluation for sequential decision-making and supporting a test-time scaling procedure for long-term returns.

Jul 16, 2026

Upload by Aurora