[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86040-en":3,"doc-seo-86040-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86040,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","Q-Learning Lab Teaching Reinforcement Learning Through Learner-Generated Trace Analysis","Reinforcement learning is often taught via the Bellman update, but undergraduates struggle to see how numeric values are computed and why particular actions are selected at each step. Q-Learning Lab is a single-file, offline, bilingual (Thai/English) browser tool for tabular Q-learning. It provides live Bellman substitution, detailed ε-greedy decision logging, and exportable transition traces. A learn–export–analyze workflow lets learners generate CSV data for curve and heatmap analysis. Validations include correctness checks, hyperparameter reproducibility, and reward-misspecification diagnosis.","arXiv :2607 . 10802v1 [ cs .CY] 12 Jul 2026  \nQ-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis  \nEkkachai Jueng  \nComputer Science Program  \nFaculty of Sciences and Liberal Arts  \nRajamangala University of Technology Isan  \nNakhon Ratchasima, Thailand  \n[ekkachai@sci. rmuti. ac. th](ekkachai@sci. rmuti. ac. th)  \nJuly 2026  \nAbstract  \nReinforcement learning is usually introduced through the Bellman update, yet the equation often remains abstract to undergraduates: they watch policy arrows converge but rarely observe how each value is computed or why an action is chosen. We present Q-Learning Lab, a singlefile, browser-based, bilingual (Thai/English) tool for teaching tabular Q-learning that requires no installation and no network connection. Beyond the usual gridworld visualization—colorcoded Q-values and policy arrows on a 5 × 5 world—the tool exposes a live Bellman-substitution panel that shows the numeric update at every step, and logs each transition, including the full pre-action Q-row, the greedy-versus-random decision under ε-greedy exploration, and wallcollision events, into an exportable trace. The central contribution is a learn–export–analyze loop: learners run their own agent, export the complete trace as CSV, and then analyze it themselves—producing learning curves, value heatmaps, and visitation maps—turning a passive demonstration into a source of learner-generated data for reflective inquiry. We validate the tool without human-subject data through three complementary evaluations: (i) correctness of the learned values and policy against a value-iteration ground truth on the identical MDP; (ii) hyperparameter sweeps (α, γ , ε) demonstrating that every pedagogical claim the tool makes is reproducible; and (iii) a reward-editing study that uses the ground-truth optimal policy to separate two behaviorally identical but diagnostically opposite failure modes—an exploration failure versus genuine reward misspecification—that a single edited reward can produce. We also situate the tool against existing gridworld visualizers via a feature comparison, describe its grounding in learning-by-doing pedagogy, and include a 50-minute lesson plan. The tool and all experiment code are openly available.  \n1 Introduction  \nReinforcement learning (RL) has moved from a specialist topic to core curriculum: it underlies game-playing agents, robotics, recommendation, and—through reinforcement learning from human feedback—the alignment of large language models. Tabular Q-learning [Watkins and Dayan(1992)] is the standard entry point, and virtually every introductory course presents the same object: agridworld, an agent, and the Bellman-style update  \nQ (s, a) ← Q(s, a) + α􀀂 r + γ max Q(s′, a′) − Q(s, a) 􀀃 . (1)  \nIn our experience teaching undergraduate machine-learning courses in Thailand, the difficulty is rarely the gridworld itself. Students readily accept that an agent wanders, collects rewards, and eventually finds a path. The difficulty is the mechanism: what exactly is inside the brackets of Eq. 1 at step 137 of episode 12? Which of the four Q-values did the agent look at before it moved, and did it exploit or explore? Why does the value of a state two cells away from the goal only start rising in the third episode? Existing teaching tools—from the classic Berkeley CS188 gridworld [DeNero and Klein(2010)] to recent browser-based playgrounds [Juliani(2025)]—answer these questions only indirectly: they display the outcome of learning (colored cells, policy arrows) while the update itself stays hidden inside the code.  \nQ-Learning Lab is built around the opposite premise: every quantity in Eq. 1 should be inspectable at every step, and the record of those steps should belong to the learner. The tool makes three design moves.  \nLive Bellman substitution. A dedicated panel re-renders Eq. 1 after every single step with the actual numbers substituted in: the observed reward r, the bootstrap term ","cbCaibI33K8ys3Va","https://ap.wps.com/l/cbCaibI33K8ys3Va","pdf",1049280,1,12,"English","en",105,"# Introduction\n## Live Bellman substitution\n## Decision-complete trace logging\n## The learn–export–analyze loop","[{\"question\":\"What problem does Q-Learning Lab target in reinforcement learning instruction?\",\"answer\":\"It addresses the gap between presenting the Bellman update mathematically and showing, step by step, how values are computed and how actions are chosen during Q-learning.\"},{\"question\":\"How does the tool make the Bellman update inspectable for learners?\",\"answer\":\"A live panel re-renders the Bellman equation after every step with substituted numeric terms, including reward, the bootstrap max over next-state Q-values, TD error, and old/new Q-values, with explicit annotations for terminal transitions and wall collisions.\"},{\"question\":\"What does learners’ learn–export–analyze loop enable?\",\"answer\":\"Learners run their own agent, export the complete decision trace as CSV, and analyze it to produce learning curves, value heatmaps, and visitation maps, turning a demonstration into learner-generated data for reflection.\"}]",1784208019,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"q-learning-lab-teaching-reinforcement-learning-through-learner-generated-trace-analysis","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/q-learning-lab-teaching-reinforcement-learning-through-learner-generated-trace-analysis/86040/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does Q-Learning Lab target in reinforcement learning instruction?","Question",{"text":75,"@type":76},"It addresses the gap between presenting the Bellman update mathematically and showing, step by step, how values are computed and how actions are chosen during Q-learning.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the tool make the Bellman update inspectable for learners?",{"text":80,"@type":76},"A live panel re-renders the Bellman equation after every step with substituted numeric terms, including reward, the bootstrap max over next-state Q-values, TD error, and old/new Q-values, with explicit annotations for terminal transitions and wall collisions.",{"name":82,"@type":73,"acceptedAnswer":83},"What does learners’ learn–export–analyze loop enable?",{"text":84,"@type":76},"Learners run their own agent, export the complete decision trace as CSV, and analyze it to produce learning curves, value heatmaps, and visitation maps, turning a demonstration into learner-generated data for reflection.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]