← All articles

An agent says it’s done. What would prove it?

· By Mamer Alomari · AI-assisted

#AIAgents · #AgentVerification · #SoftwareEngineering

A practical case for checking the finished state independently of the agent that built it.

The last line of an agent’s report is often the most tempting one: ‘Done.’ It is also the least useful evidence. A confident summary may accurately describe a file the agent wrote, a command it ran, or a deployment it attempted. None of those observations, on its own, establishes that the requested outcome exists where a reader or user can actually find it. Completion is a claim about state, not a property of the agent’s tone.

Imagine asking a coding agent to add a blog page. It creates an HTML file, runs a test, and reports success. The test might only check that the file exists. The navigation could still lead nowhere; a linked image could be missing; the mobile menu might hide the new link; the hosting configuration could point at another directory. A second pass should not ask the builder to explain its intention. It should visit the exact route and inspect what is served, then compare that result with the original request.

Start by writing a small contract before work begins. Name the artifact, the route, the environment, and the observable conditions that count as success. For a blog, that might mean a readable index at /blog, a post at its canonical slug, working links from the existing pages, a sensible mobile layout, and no change to an unrelated signup form. Also name the prohibited shortcuts: a screenshot of a local file is not proof that the hosted route works, and a deployment command returning zero is not proof that the public URL contains the new page.

Then separate three kinds of evidence. A trace tells you an action was attempted. An artifact tells you something was produced. An independently observed outcome tells you the intended state can be read back from the target. All three can be useful, but they answer different questions. A build log may establish that the build ran. It cannot establish that the browser received the right HTML after routing, caching, permissions, and publication all had their say.

The verifier needs a different vantage point, not merely a different name. Give it the acceptance contract and the target, but do not let the builder’s polished summary become the verification plan. It should read the actual files or remote object, follow the links, and try the failure paths. If it uses the same assumptions and the same narrow smoke test as the builder, the handoff adds ceremony without independence. A human reviewer, another agent, or a deterministic harness can play this role; the key is the separate observation.

For a static site, a useful check is deliberately boring. Open the index and post at their exact paths. Inspect the title, date, canonical URL, and navigation. Follow every internal link and confirm its target exists. Check that each referenced local asset is present and that the book waitlist markup and scripts were not accidentally replaced. At a narrow viewport, make sure the menu opens, has a labeled control, and still leads to the article. Test keyboard focus and the skip link. If a route fails, report that route, not a general impression that the site feels incomplete.

For a workflow with side effects, read back the destination after writing. A successful API response is a receipt for an operation, not necessarily evidence of the user-visible result. Inspect the exact record, post, or page under the right account and revision. If the destination cannot be accessed, say so and mark the outcome unverified. Do not quietly promote ‘write accepted’ into ‘published and visible.’ The distinction matters most when an agent’s work crosses a system boundary.

Long-running work makes this discipline even more important. Anthropic’s account of building long-running agents describes structured progress notes, incremental feature work, and checks that help a new session understand what has actually been completed.[1] That is an example of a continuity mechanism, not proof that any particular project uses it. The operational lesson is to preserve a short ledger: current objective, last verified change, evidence pointer, blocker, and next distinct action. A new worker can then test whether state advanced rather than simply repeating the previous probe.

A failed check is not an embarrassing footnote. It is information about the gap between the claim and the world. Record the expected condition, the observed condition, the environment, and the precise evidence. If a link is broken, fix the link and rerun the relevant checks. If the same test keeps producing the same failure without a new hypothesis, stop the loop and escalate the blocker. More tool calls are not inherently more progress.

There is a sensible limit to verification: not every task needs a separate bureaucracy. Keep the contract as small as the risk permits, automate stable checks, and reserve human attention for ambiguous or consequential outcomes. What should never be optional is the final distinction. The builder may say what it did. The verifier should say what changed. Until the requested state has been independently observed at the target, the honest status is not ‘done’; it is ‘awaiting verification.’

More writing →