Your agent did not do anything

This is an embarrassingly easy failure to ship because persuasive output looks like evidence of work. A chat surface shows the answer. A run record says “completed.” A summarizer turns the answer into “implemented retry handling.” By the time a human notices the empty diff, three layers of the system agree that the agent succeeded. They are all quoting the same prose.

The useful fix starts with five ordinary words: proposal, attempt, effect, observation, evidence.

A proposal is what the model asks to happen: apply this patch, run this command, update this record. An execution attempt is the tool invocation and whatever receipt it returns. For this repository example, an effect means the attempt-attributable, durable target-state delta at the observation checkpoint—not every transient write that may have been created and reverted. An observation is what the system can inspect afterward: a diff, an exit status, a file hash, test output, a database row. Evidence is the bounded judgment that named observations satisfy a named outcome rule. A system that must detect transient or reverted mutations needs an append-only event or filesystem trace; a final diff cannot prove that history.

The distinctions matter. A proposal rejected by policy creates no attempt and no effect. An authorized call may fail before changing anything: failed attempt, no effect. A tool may report success while a fresh repository diff remains empty: completed attempt, no repository effect. A real change may occur while its response is lost: attempt known, effect unknown until the environment is read. A fresh diff can establish an observed change, after which an evaluator may still reject the outcome because a required behavior is absent or an unrelated file changed. Collapsing these states into a boolean called success gives both tool receipts and model prose more authority than they deserve.

To make that rule concrete, consider Patchwork, a fictional repository-agent scenario created as a teaching example, with two invariants. First, model output cannot mark an effect observed or an outcome verified. Second, an observed delta is attributable to the attempt only when it is bound to the attempt's recorded before-state and isolated workspace. The controller records the repository revision and workspace identity before execution. The adapter operates under a workspace lease that excludes concurrent writers, then compares that exact workspace with the recorded revision. If the lease is lost or the before-state no longer matches, attribution becomes unknown and the run cannot certify the change. Only a bounded evaluator can compare an attributable delta with the required outcome and rules against collateral changes.

For a tiny task, that might mean:

• the model proposes adding a timeout branch to client.ts;
• the controller records the starting commit and allocates an isolated worktree to this attempt;
• policy permits writes only to client.ts and its test;
• the tool reports whether it wrote the file;
• a fresh diff from that recorded commit in that worktree shows the exact changed paths;
• the typecheck and targeted test report their own results;
• the evaluator accepts only if the required behavior changed and no forbidden path did.

If the model's final message says “all tests pass” while no test observation exists, the message changes no state label. If the write tool reports success but an attempt-bound read of the isolated worktree shows no durable delta from the recorded revision, the attempt produced no repository effect in the bounded sense above. If that read is unavailable, the lease was lost, or the before-state mismatches, attribution remains unknown rather than absent. If an attributable change is observed but the test fails—or the allowlist check finds a collateral edit—the outcome remains rejected. The prose can help diagnose the run; it cannot promote the run.

The OpenAI Agents SDK runner documentation↗ is a bounded current reference for one implementation's model/tool loop and stopping semantics. It does not establish this invariant or prove an outcome. The classic end-to-end arguments paper↗ is a candidate foundation for the narrower idea that some guarantees depend on information available at the endpoints. That does not mean intermediate checks are useless, and I am not claiming the paper describes model agents or Patchwork.

What this does not prove is that repository observations are complete evidence. Tests can encode the wrong requirement. A diff can miss generated or remote state. An evaluator can share the proposal's blind spot. The invariant only blocks one common category error: letting a description of work certify that the work occurred.

Where does your current system let prose change an effect, observation, or verification label without independent evidence?