[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85511-en":3,"doc-seo-85511-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85511,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights","Autonomous AI agents powered by large language models can perform parts of the scientific research cycle, but reliable evaluation of whether their discoveries are correct remains limited. Existing benchmarks often rely on LLM-based paper judging or single leaderboard metrics that weakly reflect scientific reasoning. FIRE-BENCH (Full-cycle Insight Rediscovery Evaluation) tests agents’ ability to rediscover established, verifiable findings by designing experiments, running them, and producing evidence-backed conclusions scored against published results.","FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights  \nZhen Wang 1 * Fan Bai 2 * Zhongyan Luo 1 * Jinyan Su 3 Kaiser Sun 2 Xinle Yu 1 Jieyuan Liu 1 Kun Zhou 1 Claire Cardie 3 Mark Dredze 2 Zhiting Hu 1 Eric P. Xing 4 5  \n Website: [https://firebench.github.io](https://firebench.github.io)  \narXiv :2602 .02905v2 [ cs .AI] 10 Jul 2026  \nAbstract  \nAutonomous AI agents powered by large language models (LLMs) are increasingly capable of running a full cycle of scientific research, yet we still lack reliable ways to verify that their discoveries are correct. Because novel findings demand costly real-world validation, existing benchmarks fall back on LLM-as-judge scoring of generated papers or single leaderboard metrics, both coarse proxies for scientific reasoning. We introduce FIRE-BENCH (Full-cycle Insight Rediscovery Evaluation), which instead asks agents to rediscover established, verifiable findings from recent, high-impact machine learning research. Given only a high-level research question from a published study, an agent must independently design experiments, run them, and draw evidence-backed conclusions, scored against the study’s documented findings. Across state-of-the-art agents with frontier backbones such as gpt-5, even the strongest reaches limited rediscovery success (\u003C50 F1), with high run-to-run variance and recurring failures in experimental design, execution, and evidence-based reasoning. Beyond diagnosing current systems, FIRE-BENCH shows that open-ended discovery can be evaluated rigorously and verifiably, laying a foundation for building reliable environments that improve agents.  \n1. Introduction  \nThe emergence of autonomous agents powered by large language models (LLMs) holds the promise of accelerating scientific discovery at an unprecedented scale. These“AI researchers” are increasingly capable of automating dis-  \n*Equal contribution 1UC San Diego 2Johns Hopkins University 3 Cornell University 4MBZUAI 5CMU. Correspondence to: Zhen Wang \u003C[zhenwang9102@gmail.com](zhenwang9102@gmail.com)>.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \ncrete stages of the research lifecycle, from literature synthesis (Zheng et al., 2025 ; Schmidgall & Moor, 2025), hypothesis generation (Baek et al., 2025 ; Si et al., 2025), to coding (Tian et al., 2024 ; Chan et al., 2025), experimentation (Kon et al., 2026), and data analysis (Majumder et al., 2025 ; Gu et al., 2024 ; Gao et al., 2025) . However, a fundamental challenge lies in rigorously evaluating their capacity for genuine scientific discovery. Validating novel outcomes often requires resource-intensive, real-world verification, such as wet-lab experiments or large-scale human expert studies, making evaluation especially difficult for agents intended to automate the full research cycle from problem formulation to empirical conclusion (Lu et al., 2024a ; Yamada et al., 2025 ; Schmidgall et al., 2025) .  \nExisting benchmarks for full-cycle research agents largely follow two evaluation paradigms. The first and more ambitious one evaluates agents for generating a complete research paper on a high-level research question (Lu et al., 2024a ; Yamada et al., 2025 ; Schmidgall et al., 2025) . While this setting is expressive, assessing the scientific validity of generated papers at scale is difficult, and many approaches rely heavily on LLM-based judging as a proxy for expert evaluation (Zheng et al., 2023 ; Schroeder & Wood-Doughty, 2024 ; Yin et al., 2026) . The second paradigm avoids subjective evaluation of papers by focusing on machine learning tasks with a single performance metric, such as improving model accuracy on a leaderboard (Huang et al., 2024b ; Chan et al., 2025 ; Wijk et al., 2025) . While objective and scalable, these benchmarks often emphasize replication (Starace et al., 2025) and provide limited insight into the broader scientific reason","cbCaisSgwJEVrrB2","https://ap.wps.com/l/cbCaisSgwJEVrrB2","pdf",3733697,1,34,"English","en",105,"# Introduction\n## Evaluation gaps in existing benchmarks\n## FIRE-BENCH approach and task design","[{\"question\":\"What problem does FIRE-BENCH address in evaluating AI agents?\",\"answer\":\"It addresses the lack of reliable ways to verify that agent discoveries are correct, since prior benchmarks often use coarse proxies like LLM-as-judge or single leaderboard metrics.\"},{\"question\":\"How does FIRE-BENCH evaluate agents differently from existing full-cycle benchmarks?\",\"answer\":\"It evaluates whether agents can rediscover established, verifiable findings from recent ML research using only a high-level research question, while withholding the original experimental details.\"},{\"question\":\"What is the observed performance of state-of-the-art agents on FIRE-BENCH?\",\"answer\":\"Across frontier backbones such as gpt-5, rediscovery success remains limited (below 50 F1) with high run-to-run variance and recurring failures in experimental design, execution, and evidence-based reasoning.\"}]",1784204090,86,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"fire-bench-evaluating-ai-agents-on-the-rediscovery-of-scientific-insights","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/fire-bench-evaluating-ai-agents-on-the-rediscovery-of-scientific-insights/85511/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does FIRE-BENCH address in evaluating AI agents?","Question",{"text":74,"@type":75},"It addresses the lack of reliable ways to verify that agent discoveries are correct, since prior benchmarks often use coarse proxies like LLM-as-judge or single leaderboard metrics.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does FIRE-BENCH evaluate agents differently from existing full-cycle benchmarks?",{"text":79,"@type":75},"It evaluates whether agents can rediscover established, verifiable findings from recent ML research using only a high-level research question, while withholding the original experimental details.",{"name":81,"@type":72,"acceptedAnswer":82},"What is the observed performance of state-of-the-art agents on FIRE-BENCH?",{"text":83,"@type":75},"Across frontier backbones such as gpt-5, rediscovery success remains limited (below 50 F1) with high run-to-run variance and recurring failures in experimental design, execution, and evidence-based reasoning.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]