CS224R Default Project Specification - RL Fine-Tuning of Language Models

Sep 11, 2026

Explore reinforcement learning post-training for large language models through a default project built around the Countdown arithmetic reasoning task using the Qwen model. Implement supervised fine-tuning for warmstarting, pairwise preference optimization with a DPO/IPO-style objective, and online policy-gradient optimization with RLOO using a rule-based verifier reward. Build a rigorous evaluation setup comparing each stage to baselines, then design a chosen algorithmic extension to stress-test improvements on downstream performance.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages23

Reading Time8 min

AccessFree to download

Initial Implementation

Background

Description of Tasks and Dataloading

Method Implementation

Evaluation Setup

Extension to the RL Methods Explored

Synthetic Data Augmentation & Pre-training Initialization

Improving Efficiency through Off-Policy Sampling

Incorporating Test Time Inference

Exploration in Discrete Token Spaces

Self-Play & Multi-Agent Co-evolution

Effective Tool-Integrated Reasoning

Learning with a Curriculum

Choose Your Own Adventure

Important Information

Project Mentors

Project Groups

Contributions

Honor Code and AI Tools Policy

Submission Format

Using Late Days

Computing Resources

Deliverables and Grading

Survey

Project Proposal + SFT Implementation

IPO + RLOO Milestone

Poster Presentation

Report

Expectation of the Default Project

| RL Fine-Tuning of Language Models
CS 224R Default Project Specification | | --- | | You will explore how to use RL for post-training large language models (LLMs) through two main parts: (1) a core implementation section where you will get the chance to implement parts of the reinforcement learning stack for LLMs, and (2) a research-oriented extension where you explore a research direction of your choosing and evaluate whether it improves downstream performance.
The default implementation is organized around the Countdown arithmetic reasoning task with the Qwen model. You will implement three core stages: (1) supervised fine-tuning (SFT) for warmstarting, (2) pairwise preference optimization using a DPO/IPO-style objective, and (3) online policy-gradient optimization with RLOO using a rule-based verifier reward. Next, you will construct an evaluation setup to rigorously compare these stages against relevant baselines. Finally, you will propose an extension of your choosing to explore how existing algorithmic ideas can be improved and stress-tested. | |
Note on Clarifications
Please use the default project Ed thread for questions and clarifications: [https://edstem.org/](https://edstem.org/)[ ](https://edstem.org/)[us/courses/97036/discussion/7926481](us/courses/97036/discussion/7926481. Course staff will use this thread to answer questions)[. Course staff will use this thread to answer questions](us/courses/97036/discussion/7926481. Course staff will use this thread to answer questions)[ ](us/courses/97036/discussion/7926481. Course staff will use this thread to answer questions)and share clarifications with students. | |

Note on default project vs. custom project:
It is not intended for the default final project to require less effort or work than the custom final project. The default project simply reduces the overall difficulty of devising your own project idea and evaluation methods, allowing students to commit an equivalent amount of effort to the provided problem. | | | |

Note on Default Project Tutorial
We will have a virtual tutorial about the default project to walk through the details of the project and the starter code. Look out for announcements regarding this on Ed. | |

Note on Implementation Integrity
Please do not use/refer to code from any existing codebase, framework, or AI agent (e.g., Cursor, Codex, Claude Code, Antigravity, etc.) for ANY part of the default project except for the final extension. You may use required infrastructure libraries such as Hugging Face for model, tokenizer, and dataset loading, but you may not use high-level trainers such as SFTTrainer for the implementation component. The Stanford Honor Code applies! | |

Note on OAE Extensions
Assignment extensions granted by the Office of OAE do not apply to group assignments. However, if you are working individually they can apply. | |
Note on Assistance
TAs may not look at students’ code for either the default or custom final projects. | | | Contents 1 Initial Implementation 4 1.1 Background ......................................... 4 1.2 Description of Tasks and Dataloading ........................... 5 1.2.1 Default Task: Math Reasoning with Countdown ................. 5 1.2.2 How Data Depends on the Training Algorithm .................. 5 1.2.3 Preference Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.4 Verifier-Based Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.5 Datasets Used in the Default Countdown Pipeline ................ 6 1.3 Method Implementation (Difficulty: Medium, Time: High) ............... 7 1.3.1 Supervised Fine-Tuning (SFT) ........................... 7 1.3.2 Preference Optimization (IPO) .......................... 7 1.3.3 Online Policy-Gradient (RLOO) .......................... 8 1.4 Evaluation Setup ...................................... 9 2 Extension to the RL

FAQs

What are the two main parts of the default project?

The project includes a core implementation section that implements parts of the RL stack for LLMs, and a research-oriented extension where you choose a research direction and evaluate its effect on downstream performance.

What stages must be implemented in the default Countdown pipeline?

You implement supervised fine-tuning (SFT), then pairwise preference optimization using a DPO/IPO-style objective, and finally online policy-gradient optimization with RLOO using a rule-based verifier reward.

How should the results of different RL stages be evaluated?

You construct an evaluation setup that rigorously compares the implemented stages against relevant baselines before proposing and testing your extension idea.

Sep 11, 2026

Uploaded by Evangeline