Reason Reward Refine Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models

Jul 16, 2026

Upload by Ezra

Physics reasoning fails structurally in small language models: a mistake at any step propagates forward and corrupts subsequent inferences. Limited domain knowledge, hallucination during multi-step derivation, and distributional sensitivity compound this issue. The framework introduces a step-level reward that locates the first reasoning error, produces targeted structured feedback, and trains revision via policy gradient with KL regularization while avoiding ground-truth solutions as generation targets. Across five physics benchmarks, it improves accuracy by 17–20% versus CoT prompting and 10–16% versus the strongest baseline, also sharply reducing calculation, miscomprehension, and conceptual misapplication errors.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages40

Reading Time1 h 41 min

Introduction

Step-Level Reward Mechanism

Error-Type Reduction via Structured Feedback

Comparison with Prior Work

Contributions and Results

Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models Raj Jaiswal 1∗, Dhruv Jain2∗, Rishabh Dhawan 1∗, Sree Krishna Uppalapati 1∗, Shin’ichi Satoh4 , Tanuja Ganu5 , Rajiv Ratn Shah3 1IIIT Delhi 2IIT BHU 3IIT Kanpur 4NII Tokyo 5Microsoft Research India {jaiswalp, rishabh23002, [sree23533}@iiitd.ac.in](sree23533}@iiitd.ac.in) [dhruv.jain.ece21@itbhu.ac.in](dhruv.jain.ece21@itbhu.ac.in) , [satoh@nii.ac.jp](satoh@nii.ac.jp) , [rajivratn@iitk.ac.in](rajivratn@iitk.ac.in) , [tanuja.ganu@microsoft.com](tanuja.ganu@microsoft.com) ∗Equal contribution arXiv :2607 .05 199v 1 [ cs .AI] 6 Jul 2026 Abstract Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17–20% over CoT prompting and 10–16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions. Code, prompts, and experimental details are provided in the Appendix section (§9) . 1 Introduction Physics reasoning is inherently sequential: each solution step depends on the correctness of all preceding steps, and an error at any point invalidates every inference that follows (Jaiswal et al., 2024 ; Anand et al., 2024 ; Ding et al., 2023) . Language models have made substantial progress on reasoning tasks (Wei et al., 2022 ; Fu et al., 2023 ; Zhang et al., 2022), yet physics reasoning remains an open challenge for small language models (SLMs) (Srivastava et al., 2025 ; Boye and Myrberg, 2025) . Larger models benefit from knowledge compression across broad training distributions; smaller models do not, and the reasoning failures that Table 1: Our Framework Achieves the Highest Average Accuracy Across All Conditions. Per-model average (%) across five benchmarks; Cell shading indicates per-row rank across conditions. Discussed in §5 . | Model | CoT | RAG | SFT | DPO | Ours | | --- | --- | --- | --- | --- | --- | | Qwen 2.5 1.5B | <br>46.1 | <br>53.3 | 41.0 | 40.7 | <br>58.5 | | LLaMA 3.2 1B | 32.5 | 41.1 | 37.5 | 37.5 | 53.9 | | LLaMA 3.2 3B | 44.8 | 51.2 | 44.0 | 44.1 | 64.4 | | Phi 3.5 Mini 3.8B | 50.5 | 54.3 | 51.0 | 52.1 | 61.5 | emerge are structurally distinct from those observed at scale (Zhang et al., 2024 ; Srivastava et al., 2025) . Prior work has progressively improved physics reasoning in language models, with each approach addressing a distinct limitation (Ding et al., 2023 ; Pang et al., 2024 ; Anand et al., 2024) . Chain-ofThought prompting (Wei et al., 2022) improves reasoning transparency by externalizing intermediate steps, though step-level error propagation across dependent steps remains unaddressed. RetrievalAugmented Generation (Lewis et al., 2020) provides access to relevant domain knowledge at inference time, though correct retrieval does not guarantee correct application of retrieved content (Anand et al., 2024) . Supervised fine-tuning on expert reasoning trajectories (Ho et al., 2022 ; Luo et al., 2023) improves structural response quality, though models trained under token prediction objectives have been shown to reproduce surface reasoning pa

FAQs

Why do errors in small language models fail to recover in physics reasoning?

Physics solutions are sequential: an incorrect step invalidates all following inferences. In small models, this propagation is amplified by limited domain knowledge and multi-step hallucination.

What is the key idea of the proposed step-level reward framework?

It identifies the first reasoning error and assigns a step-level reward, enabling the model to correct at the precise failure point rather than only optimizing complete responses.

How does structured feedback affect different error types during training?

Feedback conditioned on error type is generated during training, reducing calculation errors, problem miscomprehension, and conceptual misapplication, with the largest observed improvements reported for conceptual misapplication.

Jul 16, 2026

Upload by Ezra