Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization

Jul 16, 2026

Upload by Himbo

Scientific reasoning matters for large language models, but improving training robustness and efficiency for multi-step reasoning remains open. The work studies instruction-based molecular optimization, where answer-only SFT collapses reasoning and RLVR suffers sparse feedback from invalid or un-similar candidates. Reference-guided Policy Optimization helps but caps performance when references are weak or misaligned. Active-GRPO adaptively decides per instance when to imitate and when to reinforce, while upgrading references during training. Experiments on TOMG-Bench MOLOPT show statistically significant gains across LogP, MR, and QED.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Abstract

Introduction

arXiv :2607 .0053 1v 1 [ cs .LG] 1 Jul 2026 Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization Xuefeng Liu1∗, Mingxuan Cao2∗, Qinan Huang3 , Thomas Brettin5 , Rick L. Stevens4,5 , Le Cong1 1 School of Medicine, Stanford University 2Data Science Institute, University of Chicago 3Pritzker School of Molecular Engineering, University of Chicago 4Department of Computer Science, University of Chicago 5Argonne National Laboratory Abstract Scientific reasoning is an increasingly important capability of large language mod els, yet improving the robustness and efficiency of training such reasoning remains a key open challenge. We study this problem in instruction-based molecular op timization, where answer-only supervised fine-tuning (SFT) collapses multi-step reasoning and reinforcement learning with verifiable rewards (RLVR) suffers from sparse feedback. Reference-guided Policy Optimization (RePO) mitigates both by anchoring policy updates to dataset-provided references, but its effectiveness is tightly coupled to reference quality: weak or misaligned references impose a performance ceiling. To overcome this ceiling, we propose active reasoning, a paradigm in which the policy actively decides, on a per-instance basis, when to imitate a reference and when to reinforce its own discoveries, while continuously upgrading what it imitates. We instantiate this paradigm as Active Group Relative Policy Optimization (Active-GRPO), realized through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when the reference still outperforms the policy’s own candidates, and shifts to self-improvement via reinforcement learning once the policy has gener ated molecules that surpass the reference. The latter continuously upgrades the reference itself by replacing it with the best policy-generated candidate discovered so far, progressively raising the imitation target and ensuring that reference guid ance remains informative—rather than restrictive—throughout training. Across TOMG-Bench MOLOPT, Active-GRPO improves average SR×Sim from 0.0959 for GRPO and 0.1665 for RePO to 0.1773 under matched three-seed evaluation, with statistically significant gains on LogP, MR, and QED. 1 Introduction Large language models (LLMs) have rapidly emerged as general-purpose reasoning engines, demonstrating strong performance on tasks that demand multi-step deliberation rather than surface pattern matching [17, 38] . Through advances in chain-of-thought prompting [52], supervised fine-tuning (SFT) on reasoning traces [36, 56], and reinforcement learning with verifiable rewards (RLVR)[19, 28], modern LLMs can now solve competition-level mathematics [11, 20], write and debug complex code [9, 24], and conduct structured analyses across diverse domains. This progress has motivated a growing line of work that brings LLM reasoning to bear on scientific discovery [1, 51], where success often hinges on navigating combinatorially large hypothesis spaces under domain-specific constraints. From hypothesis generation and experimental design to candidate screening in chemistry, biology, and materials science [7, 23, 49], LLMs are increasingly positioned not as passive question ∗Equal Contribution. Correspondence to: Xuefeng Liu <xfl@stanford.edu>, Mingxuan Cao <mcao@uchicago.edu> answerers but as active reasoners that propose, evaluate, and refine scientific artifacts. Yet making such reasoning robust and sample-efficient to train remains a central open challenge—particularly in scientific domains where outputs must satisfy strict, programmatically verifiable constraints. Among these scientific reasoning tasks, instruction-based molecular optimization has emerged as a particularly demanding testbed [29, 31] . Given a source molecule and a natural-language instruction specifying desired property changes—for example, impro

FAQs

What problem does Active-GRPO address in instruction-based molecular optimization?

It targets fragile multi-step scientific reasoning training, where SFT collapses reasoning and RLVR provides sparse feedback under strict validity and similarity constraints.

How does Active-GRPO decide between imitation and reinforcement during training?

Active imitate-reinforce performs imitation learning when the reference is still better than the policy candidates, then switches to reinforcement learning once policy-generated molecules surpass the reference.

How does Active-GRPO improve the references used for guidance?

Active referencing continuously replaces the reference with the best policy-generated candidate found so far, raising the imitation target and keeping guidance informative rather than restrictive.

Jul 16, 2026

Upload by Himbo