Recovering Agent State After Sandbox Eviction or Spot Instance Interruption

Recovering agent state after an interruption is not one problem to solve but several, and treating them as interchangeable is how production runtimes end up silently discarding work. The state that matters differs across conversation history, filesystem artifacts, process memory, and external side effects. Getting those distinctions right, before choosing a checkpointing or replay mechanism, is what separates an agent platform that recovers from one that merely restarts.

Why sandbox eviction and spot interruption are not the same failure mode

Spot preemption starts with a warning. AWS Spot instances get a two-minute termination notice through the instance metadata service, and after that the instance is stopped, hibernated, or terminated outright. That window is short, but it is real, and it transforms what recovery can look like. A signal handler can flush in-memory state, commit a git checkpoint, and re-queue the job before the host disappears. The Crab checkpoint/restore runtime (arXiv:2604.28138) names spot and preemptible execution as one of its three pressing use cases precisely because the instance can be reclaimed on short notice while the right runtime preserves progress across those preemptions.

Sandbox eviction is a different failure class entirely. The sandbox process or container is terminated by the runtime scheduler itself, not by the cloud provider reclaiming a host, and the termination arrives for quota, idle timeout, memory pressure, or policy reasons. Often it is instantaneous from the agent's perspective, with no warning window at all. The distinction matters operationally because filesystem state may or may not survive depending on how the sandbox layer handles persistence. Nirvana Labs' sandbox comparison documents a persistent workspace on ABS that outlives compute while processes restart, meaning file-heavy agents can continue from committed checkpoints after process loss but RAM is gone. Modal sandboxes default to a five-minute lifetime, and Daytona separates pause, archive, and deletion into distinct lifecycle transitions. Platforms that run agents in isolated cloud sandboxes, Ellipsis among them, must design around this distinction because a sandbox eviction often arrives without the two-minute notice that spot preemption provides, narrowing the set of recovery strategies that are even possible.

The warning window is the practical dividing line. Without it, there is no drain phase, no final flush, and no commit-and-requeue. Any runtime that treats these two as the same problem will build recovery logic that fits neither.

The three state layers every agent runtime owns, and which ones survive which failures

Agent state spans at least three distinct layers, and each has a different survival profile across the two failure classes. The first layer is session and conversation history: the append-only log of model calls, tool results, and approvals. Application-level frameworks protect this layer by default. LangGraph checkpoints and the OpenAI Agents SDK session backends both write it to external stores such as PostgreSQL or Redis, where it survives both spot interruption and sandbox eviction. Dubrov's runtime guide describes this as an event table keyed by run ID, separate from checkpoints and traces, because each record proves something different.

The second layer is the filesystem and workspace: the files, installed packages, and repository state the agent has written. Durable files can survive a restart while the process, open sockets, and in-memory variables are lost, which is the key insight from the sandbox comparison literature. Git commits are the canonical checkpoint mechanism here, giving the harness a stable recovery point and the next session a compact history to inspect. Whether this layer survives sandbox eviction depends entirely on whether the provider persists the filesystem, and that answer is not uniform across providers.

The third layer is process and OS state: long-lived processes, installed services, open sockets, and in-memory variables. One published evaluation quantifies the cost: chat-only recovery achieves only 13 percent and 8 percent on Terminal-Bench for Claude-code and iFlow-cli respectively, and 8 percent on a repository-level benchmark. Process state almost never survives sandbox eviction, and it survives spot interruption only if a full OS-level checkpoint, via CRIU or an equivalent, was taken before preemption.

A production runtime must make each layer's survival profile explicit and observable. A fourth boundary cuts across all three layers: an external action that has already reached a database, an API, or a remote repository cannot be undone by restoring the sandbox. That is the idempotency problem, and it belongs in its own discussion.

Why replaying tool calls is not the same as restoring state, and what that means for external actions

A checkpoint restores the sandbox to a prior state, but it cannot undo an action that already reached an external system. Replaying tool calls without detecting prior completion produces duplicate side effects: a second database write, a second pull request, a second payment charge. Writing to a database, opening a PR, charging a payment, or sending a notification cannot be replayed blindly.

Dubrov's runtime guide makes the durable record concrete. If a spot interruption fires mid-tool-call, the agent cannot know on restart whether the call finished. The only safe designs record the outcome of every external action before moving to the next turn.

Three patterns address non-idempotent actions at recovery. Before-and-after logging records intent, then records outcome, and the harness checks the log before issuing the call again. External lookup re-queries the observable system on restart, checking whether the git push landed or the database row exists. So durable state handling and re-authentication logic belong in your security design, not just your availability design.

What Crab's research reveals about when checkpointing is necessary and when it wastes resources

But most agent turns change nothing that matters for recovery, so full per-turn checkpointing is both unnecessary and expensive. The Crab paper (arXiv:2604.28138) identifies the agent–OS semantic gap as the root cause of both bad extremes. Agent frameworks see tool calls but not their OS effects. The OS sees state changes, but it has no turn-level context to judge what matters for recovery. That gap hides massive sparsity: more than three-quarters of agent turns produce no recovery-relevant state, so most checkpoints are wasted work.

Crab bridges the gap with three components. A coordinator aligns checkpoints with turn boundaries and overlaps the work with LLM wait time, hiding checkpoint cost behind the model's own latency. A host-scoped checkpoint/restore engine coordinates checkpoint traffic across co-located sandboxes to prevent bursty I/O from overwhelming shared storage. After restore, when a stale agent replays an earlier request, the coordinator returns a synthetic response from cached history instead of forwarding it to the LLM, repeating until the agent's logical progress catches up with the checkpoint head.

The results are concrete. On shell-intensive and code-repair workloads, Crab raises recovery correctness from chat-only levels to 100 percent, cuts checkpoint traffic by up to 87 percent, and stays within 1.9 percent of fault-free execution time. At high sandbox density, full per-turn checkpointing for Claude-code slows execution substantially because host storage bandwidth is contended. Selective checkpointing is a latency requirement at scale, not just a cost optimization.

When full sandbox snapshots outperform selective checkpointing, and what that depends on

Selective checkpointing is the right default for most coding agent workloads, but it cannot capture state that lives outside the repository. One recent study implements a frontier-style search over full Daytona sandbox snapshots and finds that higher and more efficient solve rates depend mainly on progress that sits outside the repository: databases, services, and other runtime state, not code diffs alone. For those workloads, a git diff is simply the wrong artifact.

The diagnostic question is whether the state that determines meaningful progress can be reconstructed from a git diff. If it can, selective checkpointing plus git commits is sufficient. If it cannot, as with database schema migrations, background service configuration, test environment state, or installed system packages, full snapshots are needed to preserve the progress that matters. Full-snapshot search also changes recovery strategy: rather than restarting from scratch after failure, restoring and branching from an intermediate state can outperform repeated restarts. This is the Go-Explore insight applied to agent sandboxes.

The cost trade-off is real. That maps to the batch coding-agent profile: CI triage, migration runs, and multi-service integration work, where there is no interactive latency requirement but lost progress is genuinely costly. Daytona's separation of pause, archive, and deletion is instructive here, because these lifecycle transitions determine whether a full snapshot is even available as a recovery artifact.

How git commits function as the practical workspace recovery layer for most coding agent runs

For the majority of coding agent workloads, structured git commits are the most durable, auditable, and operationally simple mechanism for workspace recovery. They do not replace OS-level checkpointing, but they cover the layer that matters most when process state is not being preserved. The reason is structural: commits live in the remote, not in the sandbox filesystem, so they survive sandbox eviction by default as long as the agent pushes before each significant step.

Anthropic's harness demonstrates the practical pattern. Initializer commits plus per-feature commits, alongside a claude-progress.txt progress file, serve as the source of truth for workspace recovery. On restart, the model reads git log --oneline -20 to recover its logical position without replaying the full conversation. Each commit is a stable, inspectable recovery point that a human or the harness can validate before resuming. Commit messages written by the agent carry intent, not just diff, so a resumed session understands what was attempted, not merely what changed.

The limits are equally clear. For workloads where those matter, git commits must be combined with the fuller snapshot approach. You should define that threshold explicitly, for example by committing after every tool call that modifies files, rather than leaving it to the model's judgment.

Spot interruption handling at the infrastructure layer: draining, re-queuing, and credential expiry

The two-minute spot termination window is long enough to execute a structured drain, but only if the signal-handling infrastructure is already in place. The mandatory pattern starts with a daemon or lifecycle event listener that polls the cloud provider's metadata endpoint for termination notices. On warning, it triggers a drain command that stops accepting new jobs, flushes in-memory state, commits a git checkpoint, and re-queues the interrupted job.

Re-queuing must carry enough context to resume, not restart. AWS Fargate Spot is available for task-based workloads, but durability falls entirely on the operator. There is no built-in replay mechanism, so the design burden described here applies fully.

Credential expiry deserves attention equal to state recovery. The safe design mints credentials fresh at session start and revokes them at session end, so a resumed session mints new credentials rather than inheriting expired ones. The economics favor spot for batch coding-agent work: CI triage, migration runs, and automated code review have no interactive latency requirement, the work is restartable and idempotent when designed correctly, and a mixed fleet with an on-demand floor sized to absorb worst-case simultaneous spot drain is the practical compromise for user-facing inference.

Audit trails and observability as a recovery dependency, not an afterthought

A runtime that recovers state without logging what was lost, what was replayed, and what may have been duplicated externally is not production-grade. At every agent turn, the log has to capture every tool call and its outcome, including whether the call completed before interruption. Session resumption events must record when a new session started and what checkpoint it restored from.

The requirements for an observability standard follow the same principle. Every production request must emit an OTel trace covering tool calls, retrieval sources, guardrail decisions, and timing, with span attributes carrying the model version, prompt version, policy version, user ID, and tenant ID. Compliance obligations intersect directly with this. High-risk systems under regulations like the EU AI Act require automatic logging capability over the lifetime of the system, with deployers retaining logs for at least six months. An interrupted run that discards its trace is a compliance gap, not just an operational problem.

The audit trail is also the ground truth for the external-action problem. Without it, the re-authentication and re-execution logic has nothing to operate against. Recovery without observability is guesswork dressed as engineering.

How agent configuration as code shapes what can be checkpointed, replayed, and governed at restart

Agent configuration that lives in version-controlled files is inherently more recoverable than configuration held in mutable runtime state. Filesystem, network, and process access controls defined declaratively can be applied identically to the replacement sandbox, with no manual reconfiguration on restart. Tool definitions and MCP server configurations specify which external systems the agent can reach, and these must be version-controlled alongside the recovery checkpoint so a resumed session has the same tool surface as the interrupted one.

Claude Code settings.json, which scopes agent capabilities via enforced permissions, and CLAUDE.md files, which provide instructional context Claude reads as guidance, both belong in the repository alongside the agent's working files. A git-based workspace restore then also restores capability scope. The industry is converging on policy-as-code as the mechanism for reproducible sandbox boundaries: NVIDIA OpenShell, announced at GTC 2026 and generally available in September 2026, wraps AI agents like Claude Code and Codex in kernel-level isolation governed by declarative YAML security policies, and Microsoft Conductor uses YAML workflow definitions for multi-agent orchestration.

The governance argument follows. A recovery that reinstates a different permission set than was running at checkpoint time becomes detectable. That is the property that makes recovery governable rather than improvised.

A decision framework for choosing the right recovery strategy given a workload's state profile

No single recovery strategy is correct for all workloads. The right combination of checkpointing granularity, git-based workspace recovery, external-action logging, and spot drain logic depends on what state the workload produces and what it costs to lose or duplicate it. Long runs justify checkpoint investment, and OpenAI's Codex team reports single runs working on one task for more than 25 hours, where a crash without a checkpoint forces costly re-execution of all prior work.

The second filter is what state the workload produces. Workspace-only workloads, where progress lives in repository files, are covered by git commits plus a persistent filesystem with selective checkpointing for the rest. Process state, long-lived processes and open sockets, requires OS-level checkpoint/restore because application-level recovery cannot reconstruct it, and Crab's selective approach provides that.

The third filter is the external action profile. If all tool calls are idempotent, replay is safe and recovery effort can focus on state layers. Interactive workloads require an on-demand floor sized to absorb simultaneous spot drain.

Teams that want these semantics without building each component individually can run agents on a managed platform that makes isolated sandboxes, session logging, fresh credential minting and revocation, and audit trails the default. Ellipsis provides exactly this for Claude Code and Codex, with every session in an isolated sandbox, credentials minted fresh and revoked on completion, and full tool-call and diff logging. The recovery infrastructure is then governed rather than improvised. The teams that get this right are the ones who know which state layer they are solving for before they choose a mechanism.