Prompt-Induced Waste in Coding Agents - Reasoning, Effort, Harness Design, and End-to-End Cost - A Preregistered Benchmark with Harness Extensions

Sep 15, 2026

Uploaded by Anda

Coding-agent efficiency cannot be captured by token count or model price alone. End-to-end cost and task success jointly depend on prompt semantics, inference effort, harness policy, model choice, task difficulty, tool use, context management, and provider accounting. Controlled experiments show that prompt wording can alter reasoning and verification behavior without changing task definitions. Additional inference effort can help on difficult tasks, yet may increase cost without gains, and intervention value varies across harnesses. Efficiency is modeled as cost per successful task induced by the agent trajectory.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

AccessFree to download

Introduction

Efficiency as cost per successful task

Controllable trajectory factors: prompt, effort, harness

Experimental setup

Models, tasks, and harnesses

Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost A Preregistered Benchmark with Harness Extensions arXiv : 2608 .0 1347v6 [cs .CL] 10 Sep 2026 Sarel Weinberger PointFive [sarel.weinberger@pointfive.co](sarel.weinberger@pointfive.co) Amir Hozez PointFive [amir@pointfive.co](amir@pointfive.co) August 2026 Abstract Coding-agent efficiency cannot be characterized by token count or model price alone. End-toend cost and task success depend jointly on prompt semantics, inference effort, harness policy, model, task difficulty, tool use, context management, and provider accounting. Controlled experiments show that prompt wording can change reasoning and verification behavior without changing the task, that additional inference effort can help on difficult tasks but can also add cost without benefit, and that the value of an efficiency intervention can change when the harness changes. Prompt, effort, and harness therefore act as interacting experimental factors rather than independent controls. We model efficiency as cost per successful task induced by the agent trajectory. Token and cache counts are measurements of that trajectory, not sufficient optimization targets. Agent evaluations should therefore measure success and end-to-end cost while controlling the system variables that determine how the trajectory is produced. 1 Introduction Coding-agent cost is usually analyzed through quantities such as input tokens, output tokens, context size, cache use, or model price. These quantities are important for billing, but they do not determine how much work an agent performs. The same task can produce different numbers of reasoning steps, model turns, tool calls, repository reads, edits, tests, and retries. We therefore ask what determines the trajectory that generates those tokens and tool actions, rather than treating token price alone as the object of study. We examine three controllable parts of that trajectory. The first is prompt semantics. Instructions can change how the agent searches, branches, verifies, and decides to stop even when the task, model, harness, and nominal effort setting are fixed. The second is inference effort. More reasoning can be useful on difficult tasks, but additional effort can also add computation without improving the solution. The third is the agent harness. Harnesses determine the system prompt and tool schemas, expose or set reasoning controls, manage context, define tool and subagent behavior, and shape when the model is called again. These choices can change the effect of both the prompt and the effort setting. The prompt experiments isolate the first factor. They show that instructions which request alternative solutions, deeper analysis, or unusually strong certainty can increase reasoning or verification without a corresponding gain in task success. The mechanism is behavioral rather than lexical: a longer prompt that merely restates the task does not reproduce these effects. The relevant variable is the work requested by the instruction. The effort experiments test the opposite boundary. Reducing computation is not always beneficial. On sufficiently difficult repository-scale tasks, additional inference effort can improve success for some model–task combinations, while in other cases it raises cost without improving outcomes. Efficiency therefore cannot be defined as minimum reasoning or minimum tool use. The relevant quantity is whether additional work improves the probability of solving the task enough to justify its cost. The harness experiments add a system-level factor. Different production agent runtimes expose different fixed context, tool surfaces, turn structures, verification policies, context-management rules, and reasoning defaults. Holding the task, model, prompt, and controller logic fixed while changing the harness can materially change both baseline computation and the effect of an effort or prompt interventi

FAQs

What happens when more inference effort is added?

On sufficiently difficult repository-scale tasks, extra inference effort can improve success for some model–task combinations. For other cases, it increases computation and billed cost without improving outcomes, so efficiency cannot mean minimum effort alone.

Sep 15, 2026

Uploaded by Anda