RL Forgets! Towards Continual Policy Optimization

Jul 16, 2026

Uploaded by Ivy

Continual post-training is a key paradigm for adapting vision-language models to evolving tasks, and reinforcement learning is often preferred over supervised fine-tuning based on claims of reduced forgetting. That claim lacks robust validation because existing results rely on outdated or narrow multimodal benchmarks. This work revisits the assumption using recent diverse reasoning tasks and introduces MRCL. Experiments show standard RL still triggers severe catastrophic forgetting. The failure is traced to an objective mismatch, and Continual Policy Optimization (CPO) mitigates drift without replay while preserving, sometimes improving, pretrained capabilities.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages30

Reading Time1 h 16 min

AccessFree to download

Introduction

Continual post-training and catastrophic forgetting

Evidence gaps in existing RL evaluations

MRCL benchmark and experimental findings

Objective mismatch analysis

KL regularization evaluated on current-task data

Forgetting caused by behavioral drift on prior-task distributions

Continual Policy Optimization (CPO)

Replay-free prior-task behavioral KL objective

Sparse parameter-movement regularization

arXiv :2607 .04364v2 [ cs .LG] 13 Jul 2026 RL FORGETS! TOWARDS CONTINUAL POLICY OPTIMIZATION Mao-Lin Luo 1,2 Zhe-Xu Wang 1,2 Zi-Hao Zhou 1,2 Bo Ye 1,2,3 , Jian Zhao3,4 , Min-Ling Zhang 1,2 , Tong Wei 1,2† 1 School of Computer Science and Engineering, Southeast University, Nanjing 210096, China 2 Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China 3Zhongguancun Academy 4Zhongguancun Institute of Artificial Intelligence ABSTRACT Continual post-training is becoming a central paradigm for adapting visionlanguage models to evolving tasks. Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting. However, the belief remains insufficiently validated, as existing evidence is largely drawn from outdated or homogeneous benchmarks. We revisit this assumption under recent and diverse multimodal reasoning tasks. To this end, we introduce MRCL, a Multimodal Reasoning Continual Learning benchmark. Experiments on MRCL show that standard reinforcement learning still suffers from severe catastrophic forgetting during continual post-training. We trace this failure to an objective mismatch: the KL regularization used in common policy optimization methods is evaluated on current-task data, whereas forgetting is caused by behavioral drift on priortask distributions. To address this problem, we propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective. CPO relaxes the intractable historical KL constraint into sparse parameter-movement regularization, limiting policy drift without storing old data. Extensive experiments across multiple model scales show that CPO consistently reduces forgetting while preserving, and in some cases improving, pretrained model capabilities. On Qwen3-VL-8B, CPO reduces forgetting by 13.7% and improves pretrained capability by 7.0% . The implementation code is available at [https://github.com/MaolinLuo/CPO](https://github.com/MaolinLuo/CPO). 1 INTRODUCTION Vision-language models (VLMs) have demonstrated strong performance across a wide range of real-world tasks (Comanici et al., 2025; Achiam et al., 2023; Radford et al., 2021) . As VLMs are increasingly deployed in dynamic environments, they require continually adapt to new data and tasks. However, such a process is hindered by the fundamental challenge of catastrophic forgetting (French, 1999), where learning new tasks degrades previously acquired capabilities. Most prior work addresses this problem under supervised fine-tuning (SFT) (Gu et al., 2026a; Chen et al., 2025b; Kurniawan et al., 2024) . Recently, reinforcement learning (RL) has emerged as a promising alternative for continual post-training (Lai et al., 2025), with several studies reporting substantially less forgetting than SFT (Shenfeld et al., 2026; Zhang et al., 2026; Jin et al., 2025; Chen et al., 2025a) . These findings suggest an appealing possibility: is RL intrinsically resistant to forgetting in continual post-training? We show that the current evidence is incomplete. Existing evaluations have two blind spots that can make forgetting appear milder than it is. First, several benchmarks are old relative to the base models being adapted. For example, Lai et al. (2025) train on datasets released between 2018 and 2023 while *Equal contribution. †Corresponding author. Figure 1: Plasticity–stability Pareto frontier during continual post-training. The y-axis reports the cumulative mean accuracy improvement (CAI) on current tasks, while the x-axis reports the cumulative mean accuracy delta (CAD) on previously learned tasks. Definitions of CAI and CAD are in Appendix B. Both RL and SFT exhibit catastrophic forgetting, whereas our method preserves prior knowledge more effectively while maintaining plasticity. using Qwen2.5-VL (Bai et al.

FAQs

What problem does the paper address in continual post-training for vision-language models?

The paper addresses catastrophic forgetting, where learning new tasks during continual post-training degrades previously acquired capabilities.

Why do the authors argue that existing evidence for “RL forgets less” is incomplete?

Existing evaluations can underestimate forgetting due to outdated/homogeneous benchmarks and narrow task distributions that do not reflect modern multimodal reasoning workloads.

How does Continual Policy Optimization (CPO) reduce forgetting differently from standard policy optimization?

CPO replaces an intractable historical constraint with a replay-free framework based on a prior-task behavioral KL objective, limiting policy drift via sparse parameter-movement regularization rather than evaluating KL on current-task data.

Jul 16, 2026

Uploaded by Ivy