One Refund Requested, Three Retries Made

A customer service agent evaluates that a certain client requires a €480 refund and triggers a corresponding issue_refund action. The action runs properly but gets timed out at the 30 seconds mark because of some issue on the side of the tool server. As per orchestrator configuration, a retry process kicks off, repeating the request. At the same moment, however, the agent hosting container experiences routine redeployment and gets restarted. Upon initialization, the agent reruns its past dialog, reinterprets the complaint, makes another assessment, and starts a new tool request: identical in essence yet differing slightly in wording.

Three retries, possibly resulting in duplicate payments, occurred without violating any software logic in particular. The same scenario may apply to deployment agents restarting rollouts due to CI hook redelivery or purchasing agents sending in duplicates following failure right between PO creation and state saving stages.


“Twice” has five meanings.

An agent system duplicates tasks at multiple levels of execution, each requiring a unique method of control:

  • Duplicated request: the SDK or the proxy makes an exact duplicate of the same HTTP request.
  • Duplicated tool call: the orchestrator calls the same tool again after a delay or worker crash.
  • Repeated reasoning: after replay or context loss, the model computes another decision. It is semantically the same, but not byte-wise.
  • Replayed event: the at-least-once broker sends the trigger again, initiating a completely new run of the agent.
  • Repeated business transaction: either two separate runs, or two agents perform the same business operation.


Standard API idempotence deals only with the first case. What breaks the idempotence most often is the generation of the key within the tool. Each call generates another key UUID which is then considered by the provider as a new operation. Therefore, the first principle follows:

Principle: generate the key from the intent, not from the attempt.


Provider guarantees are more restrictive than they seem

Stripe’s idempotency API documentation is what most people use. Stripe saves the first answer to a particular key, replays it upon retrying (including 500 errors) and and returns an error if the same key is reused with different arguments. However, on API v1 keys may be deleted once they are older than 24 hours (API v2 keeps them for 30 days). If an agent waits three days for human intervention before retrying, it is too late.

IETF's Idempotency-Key header draft (revision 08, October 2026) offers good vocabulary for the contracts: 409 Conflict for the retry arriving when the original is not yet finished, 422 for a different payload using the same key, and the mandate to publish the server's expiry policy. It is just a draft, and its implementation list contains vendors who use different headers.

The Amazon Builders' Library article by Malcolm Featonby is from 2021 but is still the best rationale available. It argues against determining duplicates from parameter hashes because a caller may really request two identical EC2 instances and demands that storing the token and applying the mutation become one ACID operation. They transfer directly to agents: two identical refunds are fine, and a non-transactional key table is just a decoration.

Durable execution comes in handy in cases 2 and 3. According to Temporal's documentation, activities are always executed at least once, and completed activities are not re-executed during workflow replay. If the model call is a recorded step, replay will use the same decision instead of prompting again. However, a worker failing to complete a job after the side effect happens will cause a retry nevertheless. Durable execution moves the problem, it doesn't eliminate the need for idempotence downstream.

Lastly, the Model Context Protocol tools specify idempotentHint annotation, but consider annotations just a hint that cannot be trusted from any untrusted server. It is a declaration rather than enforcement.


Apply to all layers, but create keys only at one layer.

Assign one task per layer:

Agent layer: suggests actions and does not mint keys. The decision by the model is saved as an action before execution.

Orchestration layer: gives an intent ID and calculates the key based on tenant, workflow ID, step ID, and business context like refund:order-8812. Keys in business scope can handle case 5 too.

Tool layer: applies a key using a unique constraint and an atomic action.

Downstream services: get the key (Idempotency-Key for Stripe, ClientToken for EC2) and assign it to any newly created resources.


The guard at the tool layer handles most of these edge cases.


Python
LEASE = CALL_TIMEOUT + timedelta(seconds=30)  # must outlive the provider call


def run_tool(call, ctx, operation):
    # The key comes from the recorded intent, never from this attempt.
    key = sha256_hex(canonical_json({
        "tenant": ctx.tenant,
        "workflow_id": ctx.workflow_id,
        "step_id": call.step_id,
        "intent_id": call.intent_id,  # e.g. "refund:order-8812:ticket-5521"
    }))

    # Business arguments only. Free text such as "reason" is left out
    # here AND is never sent to the provider.
    fp = sha256_hex(canonical_json(call.business_args))

    # One atomic statement: INSERT ... ON CONFLICT (key) DO NOTHING RETURNING
    row, created = idem.claim(key=key, fingerprint=fp,
                              lease_until=now() + LEASE)
    if created:
        return execute(key, row.fencing_token, call, operation)

    if row.fingerprint != fp:
        raise PayloadMismatch(key)              # 422: new intent, new key

    if row.status == "DONE":
        return row.result

    if row.status == "FAILED":
        raise OperationFailed(row.error)

    if row.status == "IN_PROGRESS" and row.lease_until > now():
        raise Conflict(key)                     # 409: original still running

    # UNKNOWN, or IN_PROGRESS with an expired lease (the worker died).
    # Compare-and-swap: only one caller wins, and gets a new fencing token.
    token = idem.take_over(key, expected_token=row.fencing_token,
                           lease_until=now() + LEASE)
    if token is None:
        raise Conflict(key)                     # another worker took over
    return reconcile(key, token, row, call, operation)


def execute(key, token, call, operation):
    try:
        result = operation(**call.business_args, idempotency_key=key,
                           timeout=CALL_TIMEOUT)
    except DefinitiveFailure as exc:            # e.g. card declined
        idem.mark_failed(key, fencing_token=token, error=str(exc))
        raise
    except Exception:
        # Timeout, connection reset, 5xx, anything else: outcome unknown.
        idem.mark_unknown(key, fencing_token=token)
        raise
    # Fenced write: UPDATE ... WHERE key = ? AND fencing_token = ?
    idem.complete(key, result, fencing_token=token)
    return result


def reconcile(key, token, row, call, operation):
    # Look before resending: the provider's copy is the truth.
    found = operation.find_by_key(key)          # resources are tagged with key
    if found is not None:
        idem.complete(key, found, fencing_token=token)
        return found

    if now() < row.created_at + operation.key_retention:
        # The provider still remembers the key, so resending is safe.
        return execute(key, token, call, operation)

    # The key has expired and nothing was found. A resend could pay twice.
    idem.mark_unknown(key, fencing_token=token)
    raise NeedsHumanReview(key)



The fingerprint covers business arguments only; free-text fields the model can rephrase, such as “reason”, are excluded and never sent to the provider. Any error short of a definitive failure (a timeout, a dropped connection, a 5xx) moves the record to UNKNOWN, never to FAILED, because the result really is unknown. A retry that finds UNKNOWN or an expired lease takes the row over with a compare-and-swap, so only one worker reconciles, and every later write is fenced by its token. Reconciliation looks for the resource before resending, and resends with the same key only while the provider still remembers it.

Whenever the API permits it, design the operation as desired state instead of as an action. "Ensure release 2.3.1 is deployed in production" is converged while "deploy a release" is not. This leads us to our second principle:


Principle: Always prefer "ensure state" over "do action".


In the event that a duplicate occurs, the last resort would be compensation, which is a saga action which undoes the creation of the duplicate purchase order or deployment. In payments, it would be a second, visible by the customer, transaction.


Edge cases determine the success of the design.

  • Concurrent agents on the same ticket: the unique constraint chooses the winning agent; the losing agent receives a conflict and learns of the stored outcome.
  • Network timeouts:  look up the resource by its key first. Retry with the same key only while the provider still remembers it; after that, a retry could pay twice, so escalate to a human.
  • Partial failures: one key per workflow step, along with a transactional outbox for changes of state and emitted events.
  • Expired keys: your retention period should be higher than maximum workflow duration and include time spent awaiting approvals.
  • Payload modifications: reject such attempts; if you really changed your mind, create another intent ID and register the reason.
  • Restart after interruption: continue from durable log; never prompt again steps whose side effect has already been started.
  • Replaying events: remove duplicates based on event ID within the same transaction as consumer's state changes.


Checklist for reference implementation.

  1. Assign each tool to one of read-only, convergent and non-idempotent categories; validate MCP hints but do not trust them.
  2. Store every model decision with the intent ID prior to its execution.
  3. Determine keys deterministically in the orchestrator; never generate them in tools.
  4. Use a unique constraint storing the status, fingerprint, lease and result of the key.
  5. Propagate keys downstream and label resources created in each tool with the key.
  6. Timeouts should be mapped to UNKNOWN; retry operation with the same key only within the provider’s retention window.
  7. Create a reconciliation query for each non-idempotent tool.
  8. Set key retention period at or above maximum workflow duration and retry horizon.
  9. De-duplicate incoming events transactionally.
  10. Record key, intent ID, attempt count and outcome of each execution for auditing purposes.
  11. In a testing environment, introduce failures: kill the worker after the side effect, drop responses, start two agents on one ticket, redeliver triggers, modify payloads, advance the clock past key expiry. Verify exactly one business effect every time.


Costs do exist, but they are manageable.

Every side effect comes with one stored row per key life cycle, another trip to a strongly consistent store along the hot path, and reconciling code that also has to be tested. There is no need for read-only tools, naturally idempotent operations and cheap, reversible side effects like making a draft. Save the machinery for actions where duplication would cost money, availability or trust.


Ask about where the idempotency key is created.

When assessing a tool or orchestration platform based on agents, the relevant question is: “What will happen on resume after a worker crash during the side effect of a tool, but before recording the result?” For API providers, enquire into the lifetime of the keys and handling of concurrent reuse and payload changes. If the answer is vague, you're handling it in your code.