[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85029-en":3,"doc-seo-85029-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85029,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks","DeepSWE provides a benchmark of 113 original, long-horizon software engineering tasks designed to evaluate coding agents in realistic repository settings. It addresses contamination and grading-faithfulness issues seen in mining-based benchmarks by authoring tasks from scratch across 91 active open-source repositories and multiple languages, never upstream merging them. Each task is assessed by a hand-written verifier for observable functionality, reducing judge disagreement versus inherited test suites and yielding wider discrimination on frontier models.","arXiv :2607 .07946v 1 [ cs . SE] 8 Jul 2026  \nDeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks  \nWenqi Huang 1 , Charley Lee 1 , Leonard Tng 1 , and Serena Ge 1  \n1 Datacurve  \nMay 2026  \nAbstract  \nDeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one.  \nDeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE’s verifier about an order of magnitude less often than with SWE-Bench Pro’s inherited tests (1.4% versus 32.4%) . Despite being about half the length of SWE-Bench Pro’s prompts, DeepSWE’s prompts describe tasks whose reference solutions touch 5.5 × more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.  \n1 Introduction  \nLarge language models have moved from completing single functions to acting as autonomous agents that carry out multi-step engineering tasks inside real repositories: locating the relevant code, editing across files, running tests, and iterating until the change works. Benchmarks are how the field tracks this progress and decides which models to trust. To serve that role, they need realistic tasks and faithful grading.  \nMost public agentic coding benchmarks descend from SWE-bench [Jimenez et al. , 2024] and are built by mining merged fixes from public repositories. That recipe is scalable, but it carries two structural costs. The fixes and their discussion are already public and plausibly in pretraining data, so a high score can reflect recall rather than problem-solving, an effect documented directly  \nfor SWE-bench [Liang et al. , 2025] . And grading is inherited from the tests each merged patch happened to ship: those tests were written to confirm one particular fix, not to grade an arbitrary new solution, so they can reject a valid alternative implementation or accept an incomplete one. Both weaken the link between a leaderboard position and an agent’s ability to solve genuinely novel engineering problems.  \nDeepSWE is a long-horizon software engineering benchmark that targets these gaps, with four contributions relative to today’s public benchmarks:  \n• Original tasks, authored at repository scale: every task is authored from scratch and never merged upstream, so its reference solution is absent from the public commit and pull-request record that pretraining corpora scrape (§3.3) .  \n• High diversity: tasks span a broad pool of 91 repositories across five languages.  \n• Larger solutions from shorter prompts: prompts are about half the length of SWE-Bench Pro’s, yet reference solutions require 5.5 × more code and ∼2 × more output tokens.  \n• Functional verification: verifiers are hand-written to test observable software behavior rather than implementation details.  \ngpt-5.5 [xhigh]  \ngpt-5.4 [xhigh] claude-opus-4 .7 [max] claude-sonnet-4.6 [high] gemini-3.5-flash [medium] claude-opus-4 .6 [max] gpt-5.4-mini [xhigh]  \nkimi-k2.6  \nmimo-v2.5-proglm-5.1 gemi","cbCaiag0ELFWsXaB","https://ap.wps.com/l/cbCaiag0ELFWsXaB","pdf",713926,1,32,"English","en",105,"# Abstract\n# Introduction\n# Scope","[{\"question\":\"What problem does DeepSWE address in existing agentic coding benchmarks?\",\"answer\":\"DeepSWE targets two weaknesses: benchmark contamination where merged fixes may be seen during pretraining, and grading that inherits tests written for a specific merged patch instead of evaluating arbitrary correct solutions.\"},{\"question\":\"How does DeepSWE make its tasks resistant to contamination?\",\"answer\":\"Tasks are authored from scratch across 91 active open-source repositories in five languages and are never contributed upstream, so reference solutions do not appear in the public commit or pull-request records that training scrapers use.\"},{\"question\":\"How does DeepSWE grade agent outputs and why is it different from typical inherited tests?\",\"answer\":\"DeepSWE uses hand-written verifiers that check requested functionality and accept any implementation that achieves it, rather than relying on the tests shipped with a particular merged fix.\"}]",1784200498,81,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"deepswe-measuring-frontier-coding-agents-on-original-long-horizon-engineering-tasks","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/deepswe-measuring-frontier-coding-agents-on-original-long-horizon-engineering-tasks/85029/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does DeepSWE address in existing agentic coding benchmarks?","Question",{"text":74,"@type":75},"DeepSWE targets two weaknesses: benchmark contamination where merged fixes may be seen during pretraining, and grading that inherits tests written for a specific merged patch instead of evaluating arbitrary correct solutions.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does DeepSWE make its tasks resistant to contamination?",{"text":79,"@type":75},"Tasks are authored from scratch across 91 active open-source repositories in five languages and are never contributed upstream, so reference solutions do not appear in the public commit or pull-request records that training scrapers use.",{"name":81,"@type":72,"acceptedAnswer":82},"How does DeepSWE grade agent outputs and why is it different from typical inherited tests?",{"text":83,"@type":75},"DeepSWE uses hand-written verifiers that check requested functionality and accept any implementation that achieves it, rather than relying on the tests shipped with a particular merged fix.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]