The Durable Agent

What you’ll learn

Every part so far made the agent smarter, safer, or cheaper, and every one of them quietly assumed the same thing: the agent lives in a single process and that process stays alive. Pull the plug mid-run and the assumption collapses. The half-finished refund, every observation, every decision the controller made, all of it lives in RAM and dies with the process. That alone is bad, but the recovery is worse, because the obvious recovery is the dangerous one: just run the task again. If the refund already posted to the payment gateway before the crash, a naive rerun posts it a second time, and now you have charged the customer’s card twice while trying to be helpful. The agent has to survive a crash and survive its own recovery. This part makes durability real with three pieces that work together. First, an append-only event journal: every step writes an event (run_started, llm_decided, tool_result, finished) to a log keyed by a fixed run id, and the key idea is that state is the fold over the log. To know what has happened you replay the events, you never trust an in-memory variable, because that variable is exactly what the crash destroyed. Second, deterministic replay with step memoization: on resume you fold the journal, return the recorded result for any step that already finished without re-running it, and execute only the tail after the crash. Third, idempotency keys for side-effecting tools, because memoization alone cannot cover the worst-case crash, the one where the effect happened but its result was never journaled. Together they give you effectively-once execution: a refund that was halfway issued when the process died is neither lost nor double-paid.

Prerequisites

You need basic Python: functions, a dictionary, a list, and a loop. That is all. Part 2 and Part 8 help, but the essay is self-contained. Part 2 gave the refund tool a local idempotency guard, an in-memory dictionary that stops a retry inside one run from double-acting, and this part is the promised hardening of exactly that guard, so it is worth knowing where it came from. Part 8 was the last “limits” part, the one that taught an agent when to stop a runaway loop, and durability is the limit that survives a kill rather than a budget. But where this part leans on an earlier idea it restates it in a sentence, and it does not re-teach the ReAct loop or the tool contract; those are owned by Part 1 and referenced here. The companion code, durable_agent.py, runs offline with no API key, no network, and no dependencies, so you can read every line and reproduce every trace in this essay yourself. The timestamps and the run id are frozen so the journal is byte-reproducible on this deterministic path, with the real LLM path (generate()) one environment flag away, exactly as in Parts 1 through 8.

What is new since the in-memory agent

I want to be precise about what is new, because almost everything else in both series happened in RAM and never thought about it. RAG Part 19’s agent and RAG Part 20’s conversational agent were always-in-memory: state was a Python variable, the loop ran to completion or it did not, and a crash simply lost the work, end of story. Nothing in either part survived the process, because nothing was asked to. This part adds an orthogonal layer none of them had, durability, and it is orthogonal on purpose. It does not change how the loop decides, what the tools are, or how a failure is classified; it changes what survives when the process dies between two steps. The most concrete way to see it is through Part 2’s guard. That guard was a local in-memory dictionary keyed by order_id: it protected a retry inside one live process, and it died with that process. Here the same idea becomes durable. The journal records intent and result on disk before and after every step, replay reconstructs state from that journal, and a stable idempotency key handed to the payment provider makes the side effect effectively-once across a crash and a replay, not just within one run. Seed in Part 2, hardened here. I am not re-deriving the loop; I am wrapping it in a layer that assumes the floor can drop out from under it at any moment.

The journal: state is the fold over the log

Start with the log, because everything else is built on it. The world the agent operates on bundles three artifacts that must survive a crash: the journal (the append-only event log), the keystore (idempotency keys the payment provider has already honored), and the ledger (the real side effect, where money actually moved). Every step appends an event to the journal, and an event is small and total: a sequence number, the fixed run id, a frozen timestamp, a type, and a data payload. We keep it in a list here and print it as JSONL, but the mental model is a file: a real system appends each line to a JSONL file or a row to SQLite so it survives the process. The point of writing it down is the rule that follows from it. State is the fold over the log. To know which steps completed, what they returned, and whether the run finished, you replay the events from the start and fold them into state. You do not read an in-memory variable, because the whole premise of this part is that the in-memory variables are gone. Here is what survives on disk after the first run crashes, quoted from the artifact’s real output:

    {"data": {"run_id": "run-7f3a"}, "run_id": "run-7f3a", "seq": 0, "ts": "2026-07-05T10:00:00Z", "type": "run_started"}
    {"data": {"args": {"query": "refund policy window"}, "idx": 0, "tool": "search_policy"}, "run_id": "run-7f3a", "seq": 1, "ts": "2026-07-05T10:00:01Z", "type": "llm_decided"}
    {"data": {"idx": 0, "result": "Refunds after the window are refundable minus a 10% restocking fee.", "tool": "search_policy"}, "run_id": "run-7f3a", "seq": 2, "ts": "2026-07-05T10:00:02Z", "type": "tool_result"}
    {"data": {"args": {"amount": 180.0, "order_id": "ORD-3300"}, "idx": 1, "tool": "process_refund"}, "run_id": "run-7f3a", "seq": 3, "ts": "2026-07-05T10:00:03Z", "type": "llm_decided"}

Read it sequence by sequence. seq 0 is run_started, the marker that this run (run-7f3a) exists. seq 1 is llm_decided for step 0: the controller chose to call search_policy, and we recorded the decision before running it. seq 2 is the tool_result for step 0: the search returned Refunds after the window are refundable minus a 10% restocking fee., and we recorded that result. So step 0 is fully complete on disk, decision and result both. Then seq 3 is llm_decided for step 1: the controller chose to call process_refund for ORD-3300 at 180.0. And then the journal stops. There is a decision for step 1 but no tool_result for it. That gap is the entire drama of the next two sections, and it is why folding the log beats trusting a variable: the log tells you exactly how far you got and, just as importantly, exactly where it is ambiguous.

The crash that breaks naive recovery

The crash is placed at the hard spot on purpose. It is not “the process died before the refund,” which would be easy, and it is not “the process died after everything was journaled,” which would also be easy. It is the nasty middle: the refund posts to the provider and the key is honored, and then the process is killed, before the tool_result for that step ever reaches the journal. Here is the run, quoted from the artifact’s real output:

    step 0 search_policy: Refunds after the window are refundable minus a 10% restocking fee.
    step 1 process_refund: effect done (ledger touched, key honored), but
    *** PROCESS KILLED before the result was journaled ***

And here is the state it leaves behind:

  Ledger after the crash: ORD-3300=$180.00  (the refund DID post)
  Note: the journal has NO tool_result for step 1, so memoization alone would re-run it.

This is the whole trap in three lines. The ledger says ORD-3300=$180.00: the money genuinely moved, the refund is real, the customer was paid. But the journal, as we saw above, has no tool_result for step 1, only the llm_decided. So the durable record disagrees with reality: on disk, step 1 looks like it was decided but never finished. Any recovery that folds the journal and asks “which steps completed?” will conclude that step 1 did not, and will want to run it again. That is the gap memoization cannot close on its own, and the note in the trace says so out loud: memoization alone would re-run it. Now watch the dangerous recovery, the one that ignores the journal and idempotency entirely and just does the plan again, from the artifact’s real output:

    step 0 search_policy: re-ran from scratch
    step 1 process_refund: refunded $180.00 to ORD-3300 (no idempotency check!)
    ledger now: ORD-3300=$360.00  <- DOUBLE REFUND ($360.00). This is the bug.

Read what the naive restart did. It re-ran step 0 (harmless, search is read-only) and then re-ran step 1 with no idempotency check, posting a second $180.00 refund on top of the one that already went through. The ledger now reads ORD-3300=$360.00, a double refund, and the trace labels it plainly: This is the bug. The customer who was owed $180 got $360, and the only thing the agent did wrong was try to recover from a crash. This is not a contrived edge case. It is the default outcome of restarting a side-effecting job, and it is precisely why “just run it again” is the wrong instinct for anything that moves money.

Replay, memoization, and the idempotency key

Now the durable resume, the one that survives both the crash and the recovery. It starts from the same post-crash world (the ledger already has one refund, the journal has those four events) and replays. Quoted from the artifact’s real output:

    step 0 search_policy: memoized from journal -> 'Refunds after the window are refundable minus a 10% restocking fee.'
    step 1 process_refund: refunded $180.00 to ORD-3300 (idempotent: key already honored, no second charge)
    finished -> Refund of $180.00 for ORD-3300 is complete (processed exactly once).

  Ledger after the resume: ORD-3300=$180.00  <- still ONE refund. Effectively-once.

Two different mechanisms saved two different steps, and the distinction is the heart of the part. Step 0 was saved by memoization: the fold over the journal found a tool_result for step 0, so the resume returned the recorded value, memoized from journal -> 'Refunds after the window are refundable minus a 10% restocking fee.', without re-running the search. That is the easy crash, the one where the result was journaled, and memoization handles it cleanly. Step 1 is the hard crash, the one where the effect happened but the result was never journaled. Memoization is no help here, because there is no recorded result to return; the fold sees step 1 as incomplete and re-attempts it. The thing that saves step 1 is the idempotency key. The refund carries a stable key (here ORD-3300:refund:180.00), the payment provider remembers keys it has already honored in its durable keystore, and when the resume re-attempts the refund the provider recognizes the key and returns the original result instead of charging again: refunded $180.00 to ORD-3300 (idempotent: key already honored, no second charge). The run then finishes, processed exactly once, and the ledger still reads ORD-3300=$180.00. That is effectively-once: the decision to refund was re-made on resume, the effect was not. And it is the hardened form of Part 2’s local guard. Part 2’s dictionary protected a retry inside one process; this key, held by a durable keystore, protects against a re-charge across a crash and a fresh process.

One honest detail in the journal after the resume is worth pointing out, because it would otherwise look like a bug. The resumed journal grows to seq 6, and it contains two llm_decided events for step 1 (seq 3 from the original run and seq 4 from the resume) before the tool_result at seq 5 and the finished at seq 6. That second llm_decided is honest, not sloppy: on resume the agent genuinely re-makes the decision for the step it could not confirm, and the journal records that it did. What does not happen twice is the effect, because the idempotency key catches it. The journal tells the true story, “I decided to refund again, and the provider told me it was already done,” which is exactly the audit trail you want. Diagram 2 traces the key through the keystore, and the interactive figure below lets you drive the crash, the naive restart, and the durable resume side by side and watch all three artifacts (journal, keystore, ledger) move.

Open figure ↗

What we are and are not claiming

It is worth being exact about the guarantees, because durability is a domain where overclaiming gets people hurt. The journal in this part is byte-reproducible only on the deterministic path, where the plan is fixed and the timestamps and run id are frozen. With a real LLM in the loop the decisions are not byte-identical from one run to the next, so the resume does not re-derive them; instead each decision is cached in the journal when it is first made, and on replay the cached decision is read back rather than re-generated. That is best-effort faithful, not byte-for-byte, and it is the honest way to make a non-deterministic agent replayable. Second, cross-process resume here means rerunning the same script on the same journal. It is not true inter-process messaging or a live handoff between two running workers; it is the realistic model of a worker that died and a new worker that picks up the same durable log. Third, the keystore models the payment provider’s own idempotency, which is where the real guarantee lives in practice. The reason effectively-once works at all is that the provider, not the agent, is the durable authority on which keys it has honored; the agent’s job is to send a stable key and trust the provider to de-duplicate. If your provider does not support idempotency keys, this pattern degrades to “be very careful,” which is exactly the situation Part 2’s local guard left you in. The mechanism here is real; the strength of the guarantee is borrowed from a provider that takes idempotency seriously.

💡 From experience. The worst page I ever took was a payment worker that crashed mid-charge. It pulled a job off the queue, called the gateway, and the charge went through, and then the box was OOM-killed before the worker could mark the job done. The queue, doing exactly what a queue should, redelivered the job to another worker, which charged the card again. We found out from the customers, not the dashboards, and the apology emails were not fun to write. The fix was not “make the worker not crash,” because workers crash, that is the one thing you can count on. The fix was the two ideas in this part. We started writing an event for every job, decision then result, to a durable log, so a restarted worker could fold the log and know precisely how far the dead one got instead of guessing. And we put a stable idempotency key on every charge and let the gateway de-duplicate, so the redelivered job hit a key the gateway had already honored and got back the original charge instead of a new one. The first time we tested it by killing the worker on purpose right after the charge posted, the redelivery returned “already done” and the ledger did not move. That is the whole lesson: the journal tells a new process where the old one stopped, and the idempotency key makes the one step that already happened safe to attempt again. State you can rebuild and a side effect you can safely retry, and a crash becomes a non-event.

Key takeaways

• The agent has always lived in one fragile in-memory process. Kill it mid-run and the half-finished refund and all of its state are gone, and the obvious recovery, run it again, is the dangerous one: it re-charges the customer. The agent must survive a crash and survive its own recovery.
• An append-only event journal (run_started, llm_decided, tool_result, finished, keyed by the run id run-7f3a with frozen timestamps) is the source of truth, and state is the fold over the log: you replay the events to reconstruct state and never trust an in-memory variable the crash destroyed.
• The crash is placed at the hard spot: after the refund posts and the key is honored (ledger ORD-3300=$180.00) but before the tool_result is journaled, so the journal has only llm_decided for step 1. The trace says it plainly: memoization alone would re-run it.
• A naive restart that ignores the journal and idempotency re-runs both steps and posts a second refund, ledger now: ORD-3300=$360.00 <- DOUBLE REFUND ($360.00). This is the bug. That is the default outcome of blindly restarting a side-effecting job.
• The durable resume uses two mechanisms: step memoization returns step 0’s recorded result without re-running it (the easy crash), and the idempotency key lets the re-attempted refund hit a provider that has already honored the key, no second charge, so the run finishes processed exactly once and the ledger stays at $180.00. That is effectively-once, and it hardens Part 2’s local guard.
• The journal honestly records two llm_decided events for step 1 (seq 3 and seq 4): on resume the decision is re-made, but the effect is not, because the key, not the journal, de-duplicates the side effect. The decision can repeat; the charge cannot.

Glossary

• Event journal / event sourcing: an append-only log of typed events (here run_started, llm_decided, tool_result, finished) that records every step before and after it happens; the log, not any in-memory variable, is the durable source of truth, and a real system appends it to a JSONL file or SQLite so it survives the process.
• State is the fold over the log: the rule that current state is derived by replaying the journal from the start and folding the events into it, rather than read from a variable that a crash would destroy; on resume the fold tells you which steps completed, what they returned, and whether the run finished.
• Deterministic replay: re-deriving the run from the journal; byte-reproducible on the deterministic path (frozen timestamps and run id), and best-effort on the real-LLM path, where each decision is cached in the journal and read back rather than re-generated.
• Step memoization: on resume, returning the recorded tool_result for any already-completed step without re-running it, so only the tail after the crash executes; it handles the easy crash but cannot cover a step whose effect happened before its result was journaled.
• Idempotency key: a stable identifier (here ORD-3300:refund:180.00) attached to a side-effecting call so that a repeat of the same key returns the original result instead of acting again; it covers the hard crash that memoization cannot, the one where the effect landed but the result was never journaled. It hardens Part 2’s local in-memory guard.
• Keystore (provider idempotency): the durable store of keys the payment provider has already honored; it models the provider’s own idempotency, which is where the real guarantee lives, because the provider, not the agent, is the durable authority on which keys have been seen.
• Effectively-once: the guarantee that a side effect happens exactly once even across a crash and a replay; the decision may be re-made on resume, but the effect is de-duplicated by the idempotency key, so the customer is charged once, never zero times and never twice.
• Reused later: the journal built here is the substrate for Part 10 (pause and resume run on the same log) and Part 11 (spans are read out of these same events for tracing), so it is worth getting the event shape right now.

---

The rule from this part: an agent that moves money cannot live only in RAM, because a crash will eventually find it. Make the log the source of truth and treat state as the fold over it (an append-only journal of every decision and result), replay that log on resume so only the tail after the crash re-runs (step memoization), and put a stable idempotency key on every side effect so a re-attempt returns the original result instead of doubling it (effectively-once, the hardened form of Part 2’s local guard). But notice what the durable agent still cannot do. It runs to completion or it crashes and resumes, and either way it runs alone, head down, no way to stop and check. Some actions should not happen without a human saying yes (a large refund needs approval before it posts), and a user watching the run may want to correct the agent mid-flight before it commits to the wrong step. The durable agent has the journal that makes both possible, but no way yet to pause on the journal, wait for a human, and resume from exactly where it stopped. Part 10, Pause, Approve, Resume, Steer, is about that: turning the journal into a place the run can stop, hand control to a person, and pick back up without losing a thing.