Notes on systems & accountability
The timeout after the write: make an agent's retry a decision, not a reflex
A durable operation identity, a bounded retry contract, and a reconciliation path for side effects whose first response never arrived.
Picture a software-factory agent asked to publish a release note after a build passes. It sends a create request to a publishing service. The connection times out before the response arrives. The agent's next turn sees an exception, not the destination's state. If it drafts a slightly different note and calls create again, readers may see two announcements for one release. If it simply records failure, the first note may already be live. This is a hypothetical workflow, not a report of an actual incident.
A timeout answers a transport question: the caller did not receive a response in time. It does not, by itself, answer whether the remote side committed the write. AWS's discussion of retryable APIs uses the same ambiguity in an EC2 launch example: after a network timeout, the caller cannot tell from that response whether an instance is running.[1] The useful state is therefore 'outcome unknown,' not 'failed' and not 'published.' An agent that collapses those states can turn recovery into a second side effect.
The first design move is to assign the operation an identity before making the call. Store an intent record with the target account, action, canonical release identifier, payload digest, and a newly generated operation ID. The ID belongs to this one intended publication, not to the individual network attempt. If the process restarts, it reads the same record and sends the same payload with the same ID; it does not ask the model to reconstruct yesterday's intent from prose. A materially different release note is a new operation requiring its own review, not a mutation smuggled into a retry.
This only works as an automatic retry if the destination's documented contract recognizes the ID and binds it to the original operation. AWS describes a caller-provided client request identifier for idempotent APIs and cautions that inferring duplicates merely from identical parameters can be wrong: a caller may genuinely intend two identical resources.[1] Do not assume every create endpoint has that behavior. Record the endpoint's token scope, retention window, payload-mismatch behavior, and what response a duplicate returns; if those details are undocumented, stop at reconciliation rather than promising exactly-once effects.
The server side of that contract matters as much as the header name. A service that remembers the token but crashes before writing the resource can suppress a legitimate retry; one that writes the resource but loses the token can duplicate it. AWS calls out the need for the token record and related mutations to be atomic and durable.[1] If you own the API, design that boundary and test the crash points. If you do not own it, treat its stated idempotency guarantee as a bounded external dependency, not as proof that your whole multi-step workflow is atomic.
A worker can use a small explicit state machine. PREPARED means the exact approved payload and identity are durable and dispatch has not been authorized. Before touching the network, durably advance to SENT; that state means a call may have been attempted and its result is unresolved, including when the worker crashes during dispatch. CONFIRMED means a target read or documented duplicate response identifies the intended remote object and expected payload. BLOCKED means the account, payload, token contract, or remote state disagrees. These states describe evidence available to the worker; they are not a claim that every API exposes a transaction log.
After a timeout or restart from SENT, choose one of two recovery branches. When the endpoint supports a stable idempotency token for the relevant interval, retry the exact operation with the original token, bounded attempts, and backoff. When it does not, query the target with a stable correlation key or inspect the canonical destination before any new create. If the target cannot be queried reliably, escalate the unknown outcome to a human. A fresh token or a newly worded payload is not a safe default merely because the previous call threw an exception. Even a crash before the network call may leave SENT; treat it as outcome-unknown, not as permission to mint a new operation.
For a concrete observable check, imagine the operation ID 'release-note:build-482' attached to a draft for a fixed repository revision. After a simulated response loss, restart the worker from its intent record. In the read-before-retry branch, assert that no second create request is issued if the remote object already matches. In the token-retry branch, assert that the retry carries the identical operation ID and payload digest and that the service creates no second object. Then change the stored revision without changing the ID: the worker must block rather than reuse an identity for different intent. The build number is illustrative, not a claim about a real repository.
A conditional write solves a related but different race. In Google Cloud Storage, an ifGenerationMatch precondition permits a mutation only when the target has the specified generation; value 0 can constrain an object write to the case where no live object of that name exists, and a failed match returns 412.[2] That can prevent an overwrite or duplicate object name, but a 412 after a lost response does not by itself prove whether this worker created the object or a competitor did. Read the object's identity and content before treating the conflict as your success.
The check should include adverse cases, not only a clean retry. Inject a timeout before the server acts and another after it commits. Test two workers racing on the same intent, a different payload paired with the old token, an expired token window, a wrong account, and a read that returns a similarly named object created by someone else. Record which cases resolve automatically and which become BLOCKED. If the destination offers only a best-effort listing, a matching title is not enough to establish ownership of the write.
Keep the attempt ledger distinct from the publication ledger. The former can say 'request sent,' 'timed out,' and 'retried with the same operation ID.' The latter should name the exact destination object, account, payload revision, and observed final state. This distinction prevents a successful retry command from laundering an uncertain remote outcome into a completed release. It also makes a later operator's job concrete: inspect one pending identity, not replay the agent's full conversation.
The practical rule is narrow: repeat an uncertain side effect only when the operation's identity, payload, and destination contract make repetition safe. Otherwise reconcile or stop. An agent is useful for deciding what to write; the retry path should be a deterministic protocol that refuses to invent a new intention after an ambiguous commit. The next time a tool call times out, the most important question is not 'How quickly can we try again?' It is 'What fact would distinguish the first write from a second one?'