In a single-step API call, failure is simple: the call either succeeds or fails, and a retry is trivially the same as the first attempt.
In a multi-step agent workflow, failure is compound. The agent has completed steps one through seven. Step eight fails. A naive retry restarts from step one — re-running seven steps of work, burning token budget, re-querying external systems, and potentially producing different outputs from non-deterministic steps. This is not a retry. It is a full restart that wastes everything that succeeded.
Partial failure handling in agent systems requires a different pattern: checkpoint what succeeded, retry only what failed, resume from the last known good state.
Why Agents Fail Partially
Multi-step agent workflows fail partially for the same reasons distributed workflows fail partially:
Transient tool failures. A search API rate-limits. A database is briefly unavailable. A model call times out. These failures are often temporary and the correct response is a wait-and-retry, not a restart.
Worker crashes. The process running the agent is interrupted — out-of-memory, infrastructure preemption, network partition. The agent was mid-execution. Some steps completed and wrote outputs. Others were in-flight and are now lost.
Validation failures. A sub-agent's output fails a quality check. The orchestrator needs to retry that specific step with different parameters or a different model, not restart the entire workflow.
Budget exhaustion. The workflow ran out of token budget or time budget partway through. The steps that completed have value. The workflow should be resumable after budget is replenished, not discarded.
Checkpointing
Checkpointing is the practice of persisting workflow state at defined points so that execution can resume from a checkpoint rather than from the beginning.
For agent workflows, a checkpoint captures:
- Which steps have completed and their outputs
- The current state of all variables in the workflow
- The pending steps and their inputs
- Any external state that was modified (records updated, messages sent)
Checkpoints should be written after each step completes, before the next step begins. The checkpoint is the boundary between what is safe to skip on retry and what must be re-executed.
Where to store checkpoints depends on the workflow duration and criticality. For short workflows (seconds to minutes), an in-memory store with a write-through to a fast persistence layer (Redis, a database) is sufficient. For long-running workflows (hours to days), durable storage is required — the checkpoint must survive process restarts.
Per-Step Retry Policies
Not all steps should have the same retry policy. The retry strategy for a step should match the failure characteristics of that step.
Transient failures (network timeouts, rate limits): Exponential backoff with jitter. Wait before retrying to avoid thundering herd. Maximum retry count with a final fallback if all retries are exhausted.
Quality validation failures: Retry with modified parameters. If a model call produced output that failed quality validation, retrying with identical inputs often produces the same bad output. Instead, modify the prompt, adjust the temperature, or route to a different model before retrying.
Deterministic failures (invalid inputs, schema violations): Do not retry. These failures will not resolve on retry because the inputs are wrong. Surface the error and require a fix before retrying.
Idempotent vs. non-idempotent steps: Steps that can safely be re-executed (read operations, idempotent writes) can be retried aggressively. Steps that are not idempotent (sending a notification, charging a payment, deleting a record) must be tracked via checkpoint so they are not re-executed if already completed.
Lease Release on Worker Crash
A specific failure mode that requires special handling: the worker process crashes mid-step. The step was in-flight — not completed, but also not cleanly failed. The next worker to pick up the workflow needs to know: did that step complete, or is it safe to retry?
The lease pattern handles this. When a worker begins executing a step, it acquires a lease on that step — a time-bounded lock that indicates "this step is being executed." If the worker completes the step, it commits the checkpoint and releases the lease. If the worker crashes, the lease expires after a timeout, and the step becomes available for retry by a different worker.
The timeout must be longer than the step's expected execution time — a step that takes 30 seconds needs a lease longer than 30 seconds. Setting the lease timeout too short causes valid in-progress executions to be retried by a second worker, potentially producing duplicate effects.
The Non-Idempotency Problem
The hardest partial failure scenario involves non-idempotent actions: the agent sent an email, wrote to a database, or called an external API at some point in the workflow. The worker crashed. On retry, should the agent re-execute that step?
The checkpoint is the source of truth. If the checkpoint records the step as completed before the crash, the retry skips it. If the checkpoint records the step as in-flight, the retry must determine whether the action actually happened — often by querying the external system — before deciding whether to re-execute.
For high-stakes non-idempotent actions, the safest design is to record intent before execution ("I am about to send this email") and outcome after execution ("This email was sent, message ID: X") as separate checkpoint entries. A retry can then safely skip a step whose outcome was recorded, and re-execute a step whose intent was recorded but outcome was not.
LangGraph's Approach
LangGraph implements a form of this pattern natively through persistent checkpointers. Each node in a LangGraph graph writes its output to the checkpoint store before the next node executes. If execution is interrupted, the graph resumes from the last committed checkpoint rather than restarting. The built-in checkpointers support SQLite (for local development) and Postgres (for production). This is one of the reasons LangGraph is well-suited to long-running agentic workflows — the checkpointing architecture handles partial failure without requiring the developer to implement it manually.