Published field note · Research note
· 5 min read
Model leaderboards hide the harness tax
Pass rate belongs to a model, harness, environment, and budget configuration; the model name alone hides cost, latency, failure shape, and oversight.
Suppose you are choosing between two coding-agent configurations for an internal release lane. Both use MiniMax M2.5 on the same 50-task subset, and both resolve 46% of the tasks. A pass-rate sheet calls them tied.
Then operations reviews the recorded trajectories. In The Scaffold Effect v1's MiniMax M2.5 results, OpenCode uses about 1.55 million tokens per solved task and averages 2.16 no-progress steps per run. OpenHands-SDK reaches the same 46% pass rate with about 843,000 tokens per solved task and 0.66 no-progress steps. Their post-hoc failure profiles differ too: the study associates OpenCode with reasoning and time failures, while OpenHands-SDK includes more verification and maximum-turn failures. The tie was real; it was also an incomplete release decision. It does not imply that OpenHands-SDK is universally better—only that the pass-rate column did not contain the operating trade.
The operational unit of comparison is not a model name. It is a model plus harness plus environment plus budget. In this note, harness tax is shorthand for operating overhead hidden when a model-only row omits that surrounding configuration; it is not a causal marginal cost attributed to the harness alone. The harness chooses context, exposes tools, serializes observations, retries, and stops. The environment determines available state and execution conditions. The budget decides how much search, waiting, and repeated inference the system may buy. Change any of these and the label on the model row no longer identifies the system you measured.
The empirical evidence here needs prominent bounds. The Scaffold Effect v1 is preliminary work on two models—Qwen 3.6 Plus and MiniMax M2.5—across three open-source harnesses—Goose, OpenCode, and OpenHands-SDK—on a stratified 50-task subset of Terminal-Bench Pro. Those terminal-based software tasks span eight archetypes: bug fixing, build, data, implementation, machine learning, puzzle, security, and systems work; each uses a task-specific deterministic test suite. It is not a survey of all models, harnesses, coding work, or production deployments.
Within that small study, the harness-conditioned configurations differed by as much as 40 times in tokens per solved task. This is an observed configuration difference, not an isolated causal estimate of the harness alone. Paired within-model pass-rate differences were 0–8 percentage points; the 95% paired-task bootstrap intervals included zero except for the largest gap. In plain language, for nearly all observed pass-rate gaps, this sample could not rule out no underlying difference. Harness-specific failure fingerprints replicated across both models: the paper associates Goose with reasoning failures, OpenHands-SDK with verification and maximum-turn failures, and OpenCode with idle loops and time failures. Those findings are suggestive precisely because the pass-rate differences were modest while the operating behavior diverged.
The released trajectory analysis gives “no-progress” a reproducible, deliberately cheap definition: an agent step counts when it makes no tool call, or when it repeats the previous step's normalized tool and primary argument. The released MiniMax M2.5 aggregate calls this field avg_no_progress_turns; the code is counting agent steps. The displayed 2.16 and 0.66 values are the mean counts per run across the respective 50-task configurations. This is a stuck-loop proxy, not wall-clock idle time and not proof that every repeated read was useless.
The 46% in the opening is not invented: for MiniMax M2.5, the OpenCode and OpenHands-SDK configurations both reported that pass rate. The linked aggregate reports the token and no-progress values shown here, along with mutually exclusive post-hoc trial categories for reasoning, verification, time, maximum turns, hang, and infrastructure error. Turning those measurements into expected developer interruption or oversight cost is an engineering inference, not a directly measured labor study. A procurement team should measure those costs in its own lane rather than price them from this preprint.
AgentCompass v3 provides architecture evidence for keeping the variables separate. Its evaluation infrastructure defines independent Benchmark, Harness, and Environment components: benchmark owns task material and scoring, harness owns the interactive agent logic, and environment owns execution primitives and isolation. That modular design is not performance evidence for any configuration. It simply demonstrates that the separation can be made concrete in evaluation software.
The older AI Agents That Matter supplies a broader scholarly framing. It argues for cost-controlled agent evaluation, distinguishes model-development questions from downstream procurement questions, and documents reproducibility problems when agent evaluations are weakly standardized. Its lesson for a release review is not “pick the cheapest row.” It is to compare systems on the dimensions the downstream decision actually bears and to publish enough configuration detail that another team can reproduce the comparison.
So the release table needs columns beyond pass rate: model and harness versions, environment image, task set, token and wall-clock budgets, tokens per verified outcome, latency distribution, mean no-progress agent steps per run, failure class, and human interventions. A higher-cost configuration may still be the right choice if it buys a material reliability or risk improvement. A tie may conceal an obvious operational loser. Either way, the trade is visible.
There is a fair countercase. A model-only comparison is valid when harness, environment, task set, evaluator, and budget are fixed and fully matched. Under those controls, changing only the model answers a real question about the model's contribution. The error is carrying that conclusion into a deployment whose surrounding system differs.
What this does not prove: The Scaffold Effect v1 does not establish a universal 40-times harness tax, rank commercial products, or show that its failure fingerprints transfer to other settings. AgentCompass does not prove modularity improves scores, and AI Agents That Matter does not supply current procurement prices. The claim is narrower: a model name alone does not identify the measured operating system when harness, environment, or budget varies.
Which hidden harness cost would reverse your current release decision even if the headline pass rate stayed exactly the same?
Open question
What I'm testing
Which hidden harness cost would reverse your release decision even if the headline pass rate stayed exactly the same?
Evidence boundary
Sources and boundaries
A linked source supports only its named role below. This note connects and tests ideas; the note itself is not evidence.
- frontier preprintThe Scaffold Effect v1
Reports harness-conditioned token use, pass rates, and failure fingerprints for two models, three harnesses, and 50 tasks; it is not a universal harness ranking.
- empirical studyScaffold-Effects released analysis
Provides the released configurations, trial logs, aggregates, and code definition of no-progress steps behind the paper; its proxy metrics do not directly measure developer labor or production reliability.
- frontier preprintAgentCompass v3
Documents evaluation infrastructure that separates Benchmark, Harness, and Environment; this is an architecture example, not causal evidence that modularity improves performance.
- scholarly synthesisAI Agents That Matter
Argues for cost-controlled, reproducible agent evaluation and clearer procurement comparisons; it does not provide current product pricing or deployment reliability.