[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85971-en":3,"doc-seo-85971-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85971,13056703019404,"Miles","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","When Does Restricting a Coding Agent to Execute Code Help? A Regime × Agent-Design Ablation","Modern coding agents expose multiple tool surfaces—IDE primitives, bash, and Model Context Protocol (MCP) code execution—yet the field has advanced three conflicting claims about which surface matters. This work runs a crossed three-arm integrity-clean ablation (baseline, bash_only, code_only) on synthetic computation tasks and SWE-bench Mini modification tasks using two agents (Claude Code, OpenAI Codex CLI). Results show code_only is cheaper than tool-rich rivals in three benchmark-agent cells, with pass rates statistically tied; the SWE-bench/Claude exception traces to failure-cost on doomed trajectories. Implications: the cheapest tool surface depends jointly on task regime and agent design, and cost signals matter more than pass rate, which stays invariant across evaluated model sizes. Benchmark artifacts and analysis code are released at the provided repository.","When Does Restricting a Coding Agent to execute_code Help? A Regime × Agent-Design Ablation  \nHong Yang  \n[hy3134@rit.edu](hy3134@rit.edu)[ ](hy3134@rit.edu)Rochester Institute of Technology  \nRochester, New York, USA  \nQi Yu  \nRochester Institute of Technology Rochester, New York, USA  \nTravis Desell  \nRochester Institute of Technology Rochester, New York, USA  \narXiv :2607 . 10569v 1 [ cs . SE] 12 Jul 2026  \nAbstract  \nModern coding agents expose multiple tool surfaces —IDE primitives, bash, and Model Context Protocol (MCP) code-execution —and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrityclean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than — or statistically tied with — its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWEbench/Claude, where code_only is directionally costlier (+14 .4%, not significant (NS)); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost—not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at [https://github.com/hyang0129/onlycodes](https://github.com/hyang0129/onlycodes).  \n1 Introduction  \nCoding agents now ship with overlapping tool surfaces—Read, Grep, Glob, Edit, Write, and Bash on Claude Code; ACI primitives on SWEagent; an MCP execute_code tool on emerging stacks—and the field has settled on three mutually incompatible claims about which surface a coding agent should expose. Specialised IDE primitives are required: SWE-agent argues that the Agent–Computer Interface (ACI) is load-bearing, with an ACI-vs-shell ablation that produced a gain of +10 .7 percentage points on SWE-bench at the time of publication [42] . Bash alone is sufficient: mini-SWE-agent is a ∼ 100-line bash-only scaffold that posts >74% on SWE-bench Verified with no IDE primitives at all [31] . Replace the tool surface with code execution: industry write-ups report ∼98–99% token reductions by routing through an MCP execute_code tool rather than calling the native tools directly [2, 9, 32] . The three prescriptions have never been crossed on a single harness, on the same model, with regime stratification, so the field cannot tell which condition each applies under, or whether they contradict each other at all.  \nWe run the missing crossed comparison: the same harness, the same models, and the same prompts, evaluated under three tool surfaces (baseline, bash_only, code_only) crossed with two task regimes (computation, modification) and two agents (Claude Code,  \nCost ratio (code-only / cheapest rival)  \n1.3  \n1.2  \n1.1  \n1.0  \n0.9  \n0.8  \n0.7  \n\n|  | p\u003C0.001 p=0.25 |  |  |  | 1.14\u003Cbr>p=0.12 |  | p\u003C0.001 |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |\n|  |  |  | \u003Cbr>0.93 |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  | 0.80 |  |\n|  | 0.75 |  |  |  |  |  |  |  |  |\n\nArtifact Claude  \nArtifact Codex  \nSWE-bench Claude  \nSWE-bench Codex  \nFigure 1: Cost ratio (code-only / cheapest rival) per (benchmark, agent) cell, on a 1.0 parity line. The four bars are the two benchmarks × two agents that constitute the four-cell structu","cbCaisWlB1AhYW0V","https://ap.wps.com/l/cbCaisWlB1AhYW0V","pdf",637235,3,1,11,"English","en",105,"# Introduction\n## Tool-surface claims and motivation\n## Crossed comparison design\n## Experimental setup and task regimes","[{\"question\":\"What is the main question the paper investigates about coding agents?\",\"answer\":\"Whether restricting a coding agent to a single MCP execute_code tool is beneficial, and under which task regimes and agent designs this restriction helps.\"},{\"question\":\"How does the study structure the comparison across tools, regimes, and agents?\",\"answer\":\"It uses a crossed 3×2×2 design: three tool surfaces (baseline, bash_only, code_only) crossed with two task regimes (computation, modification) and two coding agents (Claude Code, OpenAI Codex CLI).\"},{\"question\":\"What do the results show regarding cost and pass rate?\",\"answer\":\"Code_only is cheaper than the cheapest tool-rich alternative in three out of four (benchmark, agent) cells, while pass rates are statistically tied within each cell; the main exception occurs in SWE-bench with Claude and is linked to failure-cost on doomed runs rather than successful-run overhead.\"}]",1784207487,28,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"when-does-restricting-a-coding-agent-to-execute-code-help-a-regime-agent-design-ablation","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/when-does-restricting-a-coding-agent-to-execute-code-help-a-regime-agent-design-ablation/85971/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main question the paper investigates about coding agents?","Question",{"text":75,"@type":76},"Whether restricting a coding agent to a single MCP execute_code tool is beneficial, and under which task regimes and agent designs this restriction helps.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the study structure the comparison across tools, regimes, and agents?",{"text":80,"@type":76},"It uses a crossed 3×2×2 design: three tool surfaces (baseline, bash_only, code_only) crossed with two task regimes (computation, modification) and two coding agents (Claude Code, OpenAI Codex CLI).",{"name":82,"@type":73,"acceptedAnswer":83},"What do the results show regarding cost and pass rate?",{"text":84,"@type":76},"Code_only is cheaper than the cheapest tool-rich alternative in three out of four (benchmark, agent) cells, while pass rates are statistically tied within each cell; the main exception occurs in SWE-bench with Claude and is linked to failure-cost on doomed runs rather than successful-run overhead.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]