Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

Jul 16, 2026

Reinforcement Learning is used to train large language models via environmental feedback, but real settings often deliver sparse or delayed signals that obscure which reasoning actions caused success or failure. A self-review step is embedded into each RL episode: after a first-pass attempt fails, the model generates a targeted review to diagnose errors and enable a better second attempt. Unlike inference-time reflection, SRRL learns the self-review with policy gradients and selectively distills improvements into the base policy, while cross-episode memory reuses effective reviews. Experiments on GSM8K with GRPO over Qwen 3-4B and OLMo-3-7B show consistently higher final reward and improved learning efficiency versus RLVR.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Introduction

Literature review

RLs and LLMs

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation Muhammad Zain Amin 1 2 Kibele Sebnem Yildirim 3 arXiv :2607 .0554 1v 1 [ cs .LG] 6 Jul 2026 Abstract Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradientsand internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement. Code github repository: [https://github.com/ZainAmin/COMP767-Project](https://github.com/ZainAmin/COMP767-Project) 1Department of Software Engineering and IT, ´Ecole de Technologie Suprieure, Montreal, Canada 2 Centre intgr de traumatologie, Hpital du Sacr-Cœur de Montral, Universit de Montral, Montreal, Canada 3Desautels Faculty of Management, McGill University, Montreal, Canada. Correspondence to: Muhammad Zain Amin <muhammad[zain.amin.1@ens.etsmtl.ca](zain.amin.1@ens.etsmtl.ca) >, Kibele Sebnem Yildirim <ki[bele.yildirim@mail.mcgill.ca](bele.yildirim@mail.mcgill.ca) >. Preprint. July 8, 2026. 1. Introduction Figure 1. The SRRL training framework These days, large language models (LLMs) are increasingly employed as decision-making agents in situations with delayed rewards and partial knowledge (Bai et al., 2026 ; Songet al., 2026 ; Wang et al., 2025 ; Zhang et al., 2025) . Reinforcement Learning provides an intuitive framework for enhancing such agents. After the model agent generatesa first-pass response, it typically receives environmental feedback in the form of outcome rewards. Environmental feedback is the information or signal the model receives from the world outside itself in response to its actions. But the problem is that the environment often provides no feedback for individual steps, making the feedback rewards delayed and sparse. Therefore, the model is confused about how to choose the correct stepwise reasoning to fix its behaviour and ends up being trained inefficiently with respect to sampling (Shi et al., 2025 ; Zhang et al., 2025) . Especially in multistep reasoning, errors in intermediate steps can invalidate the entire solution trajectories and credit assignments, leaving the model unable to recover. In supervised fine-tuning (SFT), the model’s base policy is trained by learning to reproduce human-provided fixed examples. During training, the model observes pairs of prompts and their desired responses, then adjusts its parameters to minimise the difference between the generated and desired outputs. This kind of imitation learning allows the model to become highly effective at generating highly realistic, well-structured responses it has seen before. However, this strength also creates a fundamental problem. SFT is applied to a static dataset, and the model lacks an inherent me

FAQs

Why is learning difficult when rewards are sparse or delayed in reinforcement learning for LLMs?

Environmental feedback often lacks step-level explanations, so the model cannot determine which reasoning actions led to success or failure. This makes it hard to convert each failure into meaningful behavior changes in later iterations.

How does SRRL differ from inference-time reflection methods like Reflexion?

SRRL optimizes the self-review step using policy gradients and incorporates the resulting improvements into the base policy through selective distillation. This ensures improvements persist across future episodes rather than remaining only at inference time.

What roles do cross-episode memory and policy distillation play in SRRL?

Cross-episode memory stores successful self-reviews to reuse when similar tasks appear in later episodes during training. Policy distillation internalizes improvements from self-reviews into the base policy so the gains carry over as the policy updates.

Jul 16, 2026

Upload by Theodore