[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82128-en":3,"doc-seo-82128-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82128,1099514067415,"Rowan","https://ap-avatar.wpscdn.com/avatar/100002539d78ffe74a7?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779092875211072502",8,"Research & Report","AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision","AlphaZero demonstrates neural-guided MCTS can reach superhuman performance, yet strong play may diverge from perfect, oracle-consistent play. This study compares two oracle-evaluable domains with contrasting structure: solved Connect Four and impartial Chomp characterized by Grundy-number theory. Using one self-play + MCTS pipeline, results contrast vanilla AlphaZero, a Chomp-only multi-frame variant, and AlphaZero Auxiliary Loss (AZAL) with oracle-derived policy supervision. Vanilla improves strength but loses exact optimal trajectories and invariants, while AZAL markedly increases oracle consistency across multi-seed traces and sampled states.","arXiv :2607 .08984v 1 [ cs .LG] 9 Jul 2026  \nAlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision  \nBrent Kong [bkong@caltech. edu](bkong@caltech. edu)  \nDepartment of Mathematics California Institute of Technology  \nTejas Ram [tram@caltech. edu](tram@caltech. edu)  \nDepartment of Computer Science California Institute of Technology  \nTony Yue Yu [tyy@caltech. edu](tyy@caltech. edu)  \nDepartment of Mathematics California Institute of Technology  \nAbstract  \nAlphaZero has demonstrated that a neural-guided Monte Carlo Tree Search can achieve superhuman performance, but strong play does not necessarily imply perfect play. We study this gap in two oracle-evaluable domains with contrasting structure: Connect Four, a solved partisan game with exact game-theoretic values, and Chomp, an impartial game whose optimal play is governed by Grundy-number structure. Under a unified self-play + MCTS pipeline, we compare vanilla AlphaZero, a multi-frame variant (limited to Chomp), and an AlphaZero Auxiliary Loss (AZAL) that adds oracle-derived policy supervision. We find that vanilla AlphaZero achieves strong play across both domains but cannot preserve the exact trajectories required for optimal play: in Connect Four, it fails to maintain the optimal line of play, while in Chomp, it fails to consistently restore the g = 0 invariant. On rectangular Chomp boards, multi-frame inputs alone do not remove this gap. Nevertheless, AZAL substantially improves oracle consistency across multi-seeded full-game traces and sampled-state evaluations. On Chomp, AZAL reaches perfect full-game oracle consistency on 10×11 and high but not complete consistency on 9 × 10; on Connect Four, AZAL improves oracle-match rate and delays the first oracle mistake, but does not reach perfect play.  \n1 Introduction  \nAlphaZero established the Monte Carlo Tree Search (MCTS) as a central paradigm in modern game AI. Building on the broader ideas of planning and generalization, AlphaGo Zero showed that search can be embedded directly into the reinforcement learning loop, improving policy targets while self-play supplies value supervision (Anthony et al., 2017; Silver et al., 2017) . AlphaZero then generalized this recipe across Go, chess, and shogi, illustrating that a single policy–value network guided by MCTS can achieve superhuman play from random initialization, provided only the rules of the game (Silver et al., 2018) .  \nYet superhuman play is not identical to perfect play. An AlphaZero-style system may achieve excellent results from the standard starting position while fail to choose the optimal move. This distinction has become especially important in games that depend on sparse global features and not dense local tactics. In such settings, errors in policy and value estimation do not remain isolated: they affect the search targets used to generate future self-play data. This weakens the positive feedback loop on which AlphaZero depends (Zhou & Riis, 2022; Riis, 2024) .  \nThis paper analyzes the gap between strong play and perfect play through two contrasting domains: Connect Four and Chomp. Connect Four is a solved partisan game with exact game-theoretic values, allowing direct comparison to winning trajectories (Allis, 1988) . By contrast, Chomp is an impartial combinatorial game whose winning structure is naturally studied through Grundy numbers (Sprague, 1935; Grundy, 1939) . Despite classical theoretical results, the explicit optimal strategies of Chomp remain difficult to characterize (Gale, 1974; Brouwer et al., 2005) . Together, these games provide a useful test of whether AlphaZero fails to recover exact play across games with very different strategic structure.  \nWe provide an empirical study of AlphaZero-style learning on both domains under a standard self-play + MCTS pipeline. We evaluate vanilla AlphaZero against exact oracles, compare it with a multi-frame variant inspired by Riis (2024) (limited to Chomp), and introduce an AlphaZero Auxi","cbCaicB2XcTfO1EK","https://ap.wps.com/l/cbCaicB2XcTfO1EK","pdf",2311842,1,22,"English","en",105,"# Abstract\n# 1 Introduction\n# 2 Related Work\n## 2.1 AlphaZero and expert iteration","[{\"question\":\"What gap between strong play and perfect play does the paper study?\",\"answer\":\"The paper studies cases where AlphaZero achieves strong empirical results but does not recover exact, oracle-consistent optimal play, measured via oracle evaluation.\"},{\"question\":\"Which two games are used to test AlphaZero’s ability to match exact optimal strategies?\",\"answer\":\"The paper uses Connect Four (solved, with exact game-theoretic values) and Chomp (impartial, whose optimal structure is described by Grundy numbers).\"},{\"question\":\"How do vanilla AlphaZero, a multi-frame variant, and AZAL differ in their results?\",\"answer\":\"Vanilla AlphaZero remains far from oracle-consistent play; multi-frame inputs (limited to Chomp) do not remove the rectangular-board gap; AZAL substantially improves oracle consistency but still does not guarantee perfect play on all settings.\"}]",1784178357,55,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"alphazero-in-sparsely-rewarded-games-limits-and-auxiliary-supervision","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/alphazero-in-sparsely-rewarded-games-limits-and-auxiliary-supervision/82128/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What gap between strong play and perfect play does the paper study?","Question",{"text":74,"@type":75},"The paper studies cases where AlphaZero achieves strong empirical results but does not recover exact, oracle-consistent optimal play, measured via oracle evaluation.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Which two games are used to test AlphaZero’s ability to match exact optimal strategies?",{"text":79,"@type":75},"The paper uses Connect Four (solved, with exact game-theoretic values) and Chomp (impartial, whose optimal structure is described by Grundy numbers).",{"name":81,"@type":72,"acceptedAnswer":82},"How do vanilla AlphaZero, a multi-frame variant, and AZAL differ in their results?",{"text":83,"@type":75},"Vanilla AlphaZero remains far from oracle-consistent play; multi-frame inputs (limited to Chomp) do not remove the rectangular-board gap; AZAL substantially improves oracle consistency but still does not guarantee perfect play on all settings.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]