Q-Learning Lab Teaching Reinforcement Learning Through Learner-Generated Trace Analysis

Jul 16, 2026

Reinforcement learning is often taught via the Bellman update, but undergraduates struggle to see how numeric values are computed and why particular actions are selected at each step. Q-Learning Lab is a single-file, offline, bilingual (Thai/English) browser tool for tabular Q-learning. It provides live Bellman substitution, detailed ε-greedy decision logging, and exportable transition traces. A learn–export–analyze workflow lets learners generate CSV data for curve and heatmap analysis. Validations include correctness checks, hyperparameter reproducibility, and reward-misspecification diagnosis.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Live Bellman substitution

Decision-complete trace logging

The learn–export–analyze loop

arXiv :2607 . 10802v1 [ cs .CY] 12 Jul 2026 Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis Ekkachai Jueng Computer Science Program Faculty of Sciences and Liberal Arts Rajamangala University of Technology Isan Nakhon Ratchasima, Thailand [ekkachai@sci. rmuti. ac. th](ekkachai@sci. rmuti. ac. th) July 2026 Abstract Reinforcement learning is usually introduced through the Bellman update, yet the equation often remains abstract to undergraduates: they watch policy arrows converge but rarely observe how each value is computed or why an action is chosen. We present Q-Learning Lab, a singlefile, browser-based, bilingual (Thai/English) tool for teaching tabular Q-learning that requires no installation and no network connection. Beyond the usual gridworld visualization—colorcoded Q-values and policy arrows on a 5 × 5 world—the tool exposes a live Bellman-substitution panel that shows the numeric update at every step, and logs each transition, including the full pre-action Q-row, the greedy-versus-random decision under ε-greedy exploration, and wallcollision events, into an exportable trace. The central contribution is a learn–export–analyze loop: learners run their own agent, export the complete trace as CSV, and then analyze it themselves—producing learning curves, value heatmaps, and visitation maps—turning a passive demonstration into a source of learner-generated data for reflective inquiry. We validate the tool without human-subject data through three complementary evaluations: (i) correctness of the learned values and policy against a value-iteration ground truth on the identical MDP; (ii) hyperparameter sweeps (α, γ , ε) demonstrating that every pedagogical claim the tool makes is reproducible; and (iii) a reward-editing study that uses the ground-truth optimal policy to separate two behaviorally identical but diagnostically opposite failure modes—an exploration failure versus genuine reward misspecification—that a single edited reward can produce. We also situate the tool against existing gridworld visualizers via a feature comparison, describe its grounding in learning-by-doing pedagogy, and include a 50-minute lesson plan. The tool and all experiment code are openly available. 1 Introduction Reinforcement learning (RL) has moved from a specialist topic to core curriculum: it underlies game-playing agents, robotics, recommendation, and—through reinforcement learning from human feedback—the alignment of large language models. Tabular Q-learning [Watkins and Dayan(1992)] is the standard entry point, and virtually every introductory course presents the same object: agridworld, an agent, and the Bellman-style update Q (s, a) ← Q(s, a) + α􀀂 r + γ max Q(s′, a′) − Q(s, a) 􀀃 . (1) In our experience teaching undergraduate machine-learning courses in Thailand, the difficulty is rarely the gridworld itself. Students readily accept that an agent wanders, collects rewards, and eventually finds a path. The difficulty is the mechanism: what exactly is inside the brackets of Eq. 1 at step 137 of episode 12? Which of the four Q-values did the agent look at before it moved, and did it exploit or explore? Why does the value of a state two cells away from the goal only start rising in the third episode? Existing teaching tools—from the classic Berkeley CS188 gridworld [DeNero and Klein(2010)] to recent browser-based playgrounds [Juliani(2025)]—answer these questions only indirectly: they display the outcome of learning (colored cells, policy arrows) while the update itself stays hidden inside the code. Q-Learning Lab is built around the opposite premise: every quantity in Eq. 1 should be inspectable at every step, and the record of those steps should belong to the learner. The tool makes three design moves. Live Bellman substitution. A dedicated panel re-renders Eq. 1 after every single step with the actual numbers substituted in: the observed reward r, the bootstrap term

FAQs

What problem does Q-Learning Lab target in reinforcement learning instruction?

It addresses the gap between presenting the Bellman update mathematically and showing, step by step, how values are computed and how actions are chosen during Q-learning.

How does the tool make the Bellman update inspectable for learners?

A live panel re-renders the Bellman equation after every step with substituted numeric terms, including reward, the bootstrap max over next-state Q-values, TD error, and old/new Q-values, with explicit annotations for terminal transitions and wall collisions.

What does learners’ learn–export–analyze loop enable?

Learners run their own agent, export the complete decision trace as CSV, and analyze it to produce learning curves, value heatmaps, and visitation maps, turning a demonstration into learner-generated data for reflection.

Jul 16, 2026

Upload by River Wang