[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84426-en":3,"doc-seo-84426-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84426,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Measuring AI Ability to Complete Long Software Tasks","Despite rapid progress on AI benchmarks, benchmark performance does not clearly translate to real-world capability. To quantify AI ability in human terms, the study proposes a new metric: the 50%-task-completion time horizon, defined as the time humans typically need to complete tasks where models succeed 50% of the time. Human timing is collected on RE-Bench, HCAST, and 66 shorter tasks; frontier systems like o3 show an approximately 110-minute 50% horizon, with exponential growth since 2019. Drivers include reliability, adaptation to mistakes, logical reasoning, and tool use, alongside discussions of limitations and safety implications.","arXiv :2503 . 14499v4 [ cs .AI] 10 Jul 2026  \nMeasuring AI Ability to Complete Long Software  \nTasks  \nThomas Kwa∗†, Ben West∗, Joel Becker, Amy Deng, Katharyn Garcia,  \nMax Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx  \nRyan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles‡, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler§  \nElizabeth Barnes, Lawrence Chan  \nModel Evaluation & Threat Research (METR)  \nAbstract  \nDespite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon, the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as o3 have a 50% time horizon of around 110 minutes. Furthermore, frontier AI time horizon has doubled approximately every seven months since 2019, though the trend may have accelerated since 2024 .  \nThe increase in AI models’ time horizons seems to be primarily driven by greater reliability, ability to adapt to mistakes, logical reasoning, and capacity for tool use. We discuss the limitations of our results—including their degree of external validity—and the implications of increased autonomy for dangerous capabilities.  \nIf these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.  \n1 Introduction  \nIn the last five years, frontier AI systems have undergone a dramatic transformation in capabilities, evolving from basic text generation [1] to autonomously executing complex multi-hour machine learning research projects [2] . Sufficiently capable AIs could perform dangerous, highly complex actions like autonomous development of chemical, biological, radiological or nuclear weapons (CBRN) and self-replication and adaptation outside human control [3] . Understanding AI capabilities helps inform the development of safety guardrails as systems become increasingly powerful. In particular, many frontier AI developers have committed to using measures of specific AI capabilities to determine the necessary risk mitigations for their frontier AI systems. Robust benchmarks that can accurately track and forecast AI capabilities thus form the foundation for responsible AI governance and risk mitigation.  \n∗Equal contribution.  \n†Corresponding author, [thomas.kwa@metr.org](thomas.kwa@metr.org).‡Ohm Chip. Work done at METR.  \n§Anthropic. Work done at METR.  \nPreprint. Under review.  \nFigure 1: The length of tasks (measured by how long they take human professionals) that generalist autonomous frontier model agents can complete with 50% reliability has been doubling approximately every 7 months for the last 6 years (Section 3) . The shaded region represents 95% CI calculated by hierarchical bootstrap over task families, tasks, and task attempts. Even if the absolute measurements are off by a factor of 10, the trend predicts that in under a decade we will see AI agents that can independently complete a large fraction of software tasks that currently take humans days or weeks (Section 5) .  \nHowever, existing benchmarks face several key limitations. First, they often consist of artificial rather than economically valuable tasks. Second, benchmarks are often adversarially selected fortasks that current models struggle with compared to humans, 1 biasing the comparison to human performance. Most critically, individual benchmarks saturate increasingly quickly [6], and we lack a more general, intuitive, and quantitative way to compare between different benchmarks,2 which pr","cbCaig48XdZQwJF3","https://ap.wps.com/l/cbCaig48XdZQwJF3","pdf",6195770,1,45,"English","en",105,"# Abstract\n# Introduction\n## Motivation and limitations of existing benchmarks\n## Proposed metric: X%-task-completion time horizon\n# Methodology\n## Datasets and tasks\n## Human baselines\n## Model evaluation across years\n# Results\n## 50% time horizon growth\n## Comparisons and exploratory findings\n# Limitations and implications\n# Safety and autonomy risks","[{\"question\":\"What problem does the paper address about current AI benchmarks?\",\"answer\":\"It argues that benchmark scores do not reliably indicate real-world meaning of AI capability. Existing benchmarks may be artificial, adversarially selected, and saturate quickly, making comparisons across models and tasks difficult.\"},{\"question\":\"How is the proposed metric defined?\",\"answer\":\"The paper introduces the 50%-task-completion time horizon: the task duration that humans typically take for tasks where an AI model achieves 50% success. More generally, it measures an X%-task-completion time horizon for X% success.\"},{\"question\":\"What do the results show about frontier AI progress over time?\",\"answer\":\"On the evaluated software and research-related tasks, the 50% time horizon grows exponentially from 2019 to 2025, with an approximate doubling every seven months. The growth rate after 2023 is reported to be about 20% faster than the earlier period.\"}]",1784195561,113,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"measuring-ai-ability-to-complete-long-software-tasks","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/measuring-ai-ability-to-complete-long-software-tasks/84426/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address about current AI benchmarks?","Question",{"text":75,"@type":76},"It argues that benchmark scores do not reliably indicate real-world meaning of AI capability. Existing benchmarks may be artificial, adversarially selected, and saturate quickly, making comparisons across models and tasks difficult.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is the proposed metric defined?",{"text":80,"@type":76},"The paper introduces the 50%-task-completion time horizon: the task duration that humans typically take for tasks where an AI model achieves 50% success. More generally, it measures an X%-task-completion time horizon for X% success.",{"name":82,"@type":73,"acceptedAnswer":83},"What do the results show about frontier AI progress over time?",{"text":84,"@type":76},"On the evaluated software and research-related tasks, the 50% time horizon grows exponentially from 2019 to 2025, with an approximate doubling every seven months. The growth rate after 2023 is reported to be about 20% faster than the earlier period.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]