Detecting and Stopping Runaway Agent Loops in Production

A runaway agent loop is a production failure with a specific mechanic: an agent keeps calling tools, retrying plans, or bouncing requests to another agent without making measurable progress toward its goal, and nothing in the system stops it. Fixing this after the fact, through better monitoring alone, misses the point. What's required is layered detection signals and hard termination controls that halt execution before the next API call completes, not after the invoice arrives.

Start with what a healthy agent loop actually does. Each iteration follows a perceive-plan-act-observe cycle: the agent takes in the current state, decides on a next step, executes a tool call, and reads the result before deciding what to do next. That cycle is supposed to terminate when the goal is met or when it becomes clear the goal cannot be met. A loop goes runaway when that termination never happens: the agent keeps planning, retrieving, or retrying without a valid stop condition, and without anyone building in a way to detect that no forward motion is occurring.

Production is where this breaks in ways staging never shows. Pipelines that pass every test suite fail once real-world inputs stop matching training assumptions, once a downstream tool starts returning malformed data, or once two agents begin passing requests back and forth with neither one able to break the exchange on its own. Add to that a compounding factor documented in evaluations of frontier models have found that each iteration appends more to what the agent carries forward as context, and as that accumulated context grows, model reliability drops measurably, even on simple tasks. A loop does not just fail to terminate; it degrades while it runs.

Three proximate causes explain most of these failures. First, stop conditions are missing or too loose to catch the actual failure mode. Second, systems distinguish identical retries (same call, same arguments) from equivalent retries (same intent, slightly reworded arguments) poorly or not at all, so a loop that keeps trying the same unsolvable thing in different words sails past exact-match debouncing. Third, multi-agent coordination produces ping-pong patterns where each agent is behaving reasonably given its own inputs, but the pair together is stuck. None of this is abstract. Understanding these three mechanics is what determines which monitoring signals actually mean something and which are just noise.

How far a loop can run before anyone notices (the $47K case)

In November 2025, a market research pipeline built on four LangChain agents coordinating through the A2A protocol worked exactly as designed in testing. In production, an Analyzer agent and a Verifier agent started passing requests back and forth over a schema disagreement neither could resolve alone. The loop ran for 11 days. It generated a $47,000 bill before anyone shut it down.

Nobody caught it because every individual signal looked fine. Status messages read "schema drift resolution in progress," which was, technically, an accurate description of what was happening; health checks passed because the agents were responsive and not throwing errors. The only budget alarm in the system was a monthly, account-level spend threshold, not a per-agent or per-run ceiling, so the loop's ongoing cost never crossed a line anyone had drawn. There were no step limits. No duration limits. No circuit breakers. And the loop detection that did exist checked for identical retries, not equivalent ones, so the reworded back-and-forth between Analyzer and Verifier never tripped it.

The forensic finding that came out of the post-mortem is the sharpest part of this story: a circuit breaker tied to a duration limit, set at some reasonable multiple of the task's baseline run time, would have halted the agent far sooner. The 11-day runtime traced back to an infrastructure gap, plain and simple, one that a single missing control produced.

What this case shows, that a general description of "runaway loops" cannot, is that blast radius is a function of detection time, not just of how badly a loop is behaving. A loop that runs for ten minutes before getting caught costs ten minutes' worth of tokens. A loop that runs for 11 days costs 11 days, and the difference between those two outcomes lives entirely in the instrumentation. The rest of this piece is about closing that gap before the meter runs.

The signals that indicate a loop is forming

Three signal classes matter here, and they catch different things.

Behavioral signals track whether tool calls are repeating with identical or near-identical arguments in a short window. Progress signals track whether new information is actually coming back across iterations, or whether the agent is stuck retrieving the same non-answer. Cost signals track whether token consumption is diverging from the expected baseline for a task of this type.

Each one, alone, misses something important. Behavioral signals built on exact-match debouncing miss the reworded retry, the call that changes a parameter slightly while pursuing the same dead end. Progress signals only work if "progress" has been defined ahead of time, at design time, which most teams skip because it requires deciding, task by task, what forward motion looks like. Cost signals are lagging by construction: spend is computed after tokens are returned, so a cost alert firing means the damage already happened.

The useful case is when all three arrive together. A tool call repeating, no new information across the last several iterations, and a token burn rate climbing above baseline, all at once, is close to diagnostic. Any one of the three in isolation has a dozen innocent explanations. In multi-agent systems there is a fourth pattern worth watching for specifically: one agent's output becoming another agent's input with no external state change between rounds. That is the ping-pong signature, and it is exactly what sank the Analyzer-Verifier pair for 11 days. Detection instrumentation needs to cover all three signal classes together, plus this fourth pattern in any system with more than one agent talking to another.

The instrumentation needed to catch loops early

A trace worth anything needs to capture the reasoning trace, the tools considered versus the tools actually called, the arguments passed, the responses returned, the tokens spent at each step, and the latency of each hop, all stitched into one trace that can be replayed end to end. Anything less leaves gaps exactly where loops hide.

Traditional observability tools were built for deterministic systems, where the same input produces the same log every time. Agent behavior emerges from non-deterministic model output interacting with tools and a changing environment, which produces trajectories that are genuinely hard to trace, reproduce, or explain after the fact. Worse, agents fail in ways that look like success on a standard dashboard: a well-formed but wrong answer, a redundant tool call that returns valid data, a semantically invalid action that still executes without error. None of that trips a conventional alert.

OpenTelemetry has become the emerging standard for closing this gap. Its GenAI-specific semantic conventions define span types like create_agent, invoke_agent, and execute_tool, giving teams a shared vocabulary instead of every platform inventing its own schema. More importantly for multi-agent systems, distributed trace context propagation connects spans across agent boundaries, which is exactly what makes a ping-pong pattern visible as one continuous session rather than four unrelated logs. Convergence on OTel accelerated through 2025 and into 2026, and trajectory evaluation, meaning session-level monitoring of the full sequence of tool calls rather than logging each call in isolation, is now treated as table stakes rather than a nice-to-have.

Cost observability needs to live at the span level too, not just the session level. Identifying which specific step in an agent's trajectory consumed a disproportionate share of the context window matters, because a single poorly scoped retrieval call can inflate costs by an order of magnitude once it runs at scale across many sessions.

Even good instrumentation has a gap it has not closed: tool calls are observable, but the reasoning that produced the decision to make that call is not. Trajectory-level tracing narrows this gap considerably, but it does not eliminate it. What loop detection looks like in practice, given all this: hash each tool-plus-argument call, flag when the same hash reappears within a recent window, and surface near-duplicate sequences (same intent, different wording) for a human to look at rather than trusting exact-match logic to catch them. Platforms like Langfuse, Arize Phoenix, LangSmith, AgentOps, Maxim AI, and Braintrust each approach this differently, but the criterion that matters when picking one is whether it supports session-level trajectory evaluation, not just per-call logging.

Hard termination controls (the kill switches that actually stop a loop)

The $47K case makes one distinction concrete: monitoring that fires after spend has already happened produces a cost report, nothing more. A kill switch fires before the next API call completes. Those are different categories of control, and conflating them is how an 11-day loop happens.

Maximum iteration limits are the first backstop, a hard ceiling on steps so that a loop physically cannot run forever. But this is a last resort, not real protection; by the time an iteration ceiling trips, plenty of budget is already gone.

No-progress detection catches things earlier. Hash each tool-plus-argument call and terminate execution when the same call repeats inside the detection window. Pair that with a goal-achievement check that runs before each new iteration starts, evaluating whether the task's actual objective has already been met.

Circuit breakers tied to duration matter specifically because of what the $47K post-mortem found: a task with a known baseline run time should trip a breaker at some defined multiple of that baseline, and a task expected to take fifteen minutes running for hours with no alert is the exact failure this control is built to prevent.

Per-agent token budgets, enforced before the next call goes out, stop execution rather than notifying someone that execution already cost money. That requires a shared budget registry tracking consumption in real time across agent sessions, rather than a single number that only rolls up at the account level once a month.

Cost circuit breakers evaluate spend rate before each API call and halt when a per-session or per-run ceiling is hit. The practical target is a three-layer structure: a per-agent token budget tracked in a shared registry, a cost circuit breaker checking spend rate before each call, and an enforcement gate that actually halts execution once the ceiling is reached. In that structure, it is the registry that catches the problem, not the billing dashboard.

At the implementation level, lifecycle hooks are the mechanism that does this work day to day. A DebounceHook paired with a clearly defined SUCCESS state lets a system break a loop at the reasoning layer, before a circuit breaker even needs to engage. Hierarchical budget controls close the structural hole underneath all of this: raw provider API keys have no native concept of per-consumer limits, they just produce one account-level bill with no way to cap what an individual team or agent spends. Virtual keys, as implemented in tools like LiteLLM, fix that by storing per-session iteration caps and per-session budget caps directly in key metadata.

Why the accountability gap makes these controls harder to enforce than they should be

The technical controls above are not hard to build. Getting an organization to actually own them is the harder problem, and it is structural, not technical. Cloud billing usually sits with one team, application ownership with another, and AI policy with a third, and each one tends to assume someone else has this covered. Security assumes platform engineering owns loop controls; application owners assume the model provider or a procurement team is responsible for spend limits. Nobody is wrong, exactly. Nobody is actually holding it either.

In practice, that gap produces delayed alerting, no budget guardrails at the level where they'd actually help, and no clear escalation path once a workflow starts looping. The attribution problem underneath this is mechanical: cost gets computed from token counts in a response, so a runaway agent is only visible after those tokens are already spent, and that spend rolls up at the account level by default, not attributed to the specific agent or session that generated it.

Shared credentials make the whole picture worse. Most enterprise identity and access management systems have no way to represent an AI agent as a distinct, accountable non-human identity at all. Common practice — shared service account credentials, static API keys, or tokens that impersonate a human user — forfeits exactly the accountability chain that effective spend governance depends on. If nobody can say which agent, running under whose credential, at what point in time, none of the kill switches above have an owner. The controls only work if someone is unambiguously responsible for enforcing them before a loop ever starts, not after.

Governance infrastructure that makes loop controls durable across the organization

Fixing accountability at the org-chart level requires infrastructure, not just a memo about ownership. Agent definitions, including step limits, budget ceilings, tool permissions, and circuit breaker thresholds, belong in version-controlled config files that get reviewed the way any other code change gets reviewed, not set ad hoc by whoever happens to be deploying that day. Platforms such as Ellipsis, a cloud runtime for YAML-defined coding agents, treat this as a hard constraint rather than a suggestion.

Credentials should be scoped per session, minted fresh when an agent session starts and revoked the moment it completes. That keeps a runaway loop operating with the minimum permissions it needs for the minimum time it should run, which limits blast radius well beyond just cost.

Audit trails are the enforcement record that makes any of this defensible after an incident: immutable logs of every tool call, every diff, every token spent, attributable to a specific person or trigger. This is not optional anymore in many jurisdictions. The EU AI Act's enforcement provisions, which took effect in August 2026, create explicit traceability, logging, and human oversight requirements for high-risk AI systems, and the trace data described earlier in this piece maps directly onto the audit-trail obligations laid out in Articles 12, 19, and 72.

NIST's AI Agent Standards Initiative, formally launched in February 2026, is a signal that federal standards for autonomous systems are being actively built right now. Its five topic areas, threat identification, lifecycle security, gaps in existing cybersecurity frameworks, security measurement, and environmental controls, collectively sketch what governance programs will eventually need to prove they have in place. Per IANS Research, roughly half of large enterprises had established dedicated AI governance committees as of early 2026, which means the organizational muscle to actually enforce these controls is still being built in the other half.

A layered framework for putting this into practice

No single control would have saved the $47,000 case. Any one of several missing layers, a duration circuit breaker, a per-agent budget, no-progress detection, would have changed the outcome on its own. That is the argument for a layered framework rather than a single silver-bullet fix.

Layer one is design-time, before the agent ever runs. Define explicit stop conditions: a goal-achievement check, a maximum iteration limit, a duration limit set relative to the task's expected baseline. Set per-session token budgets in version-controlled config, not as a runtime parameter someone can quietly override under deadline pressure. Scope credentials to the session and revoke them the moment it ends.

Layer two is runtime detection, while the agent is actually running. Hash tool-plus-argument calls and flag repeats inside the detection window. Check spend rate before each API call goes out, not after the response comes back. Propagate trace context across agent boundaries so a ping-pong pattern between two agents shows up as one session trace instead of two logs that look unrelated.

Layer three is enforcement, what happens once detection actually fires. A budget registry halts the session before the next call goes out, not after the bill arrives. A circuit breaker trips on duration or spend rate and escalates to a human instead of quietly retrying. An immutable log captures the full trajectory up to the point of termination, for whoever has to reconstruct what happened afterward.

Layer four is organizational accountability, meaning who owns enforcing all of the above. Ownership of each control needs to be unambiguous before an agent ever reaches production, not sorted out after an incident. Configuration gets reviewed and versioned like any other code. Governance structure needs to map onto the regulatory requirements already in force, rather than treating compliance as a separate workstream bolted on afterward.

The order across these layers is not arbitrary. Design-time controls reduce how often runtime detection ever has to fire. Runtime detection reduces how often an enforcement gate actually trips. Enforcement gates reduce how often a human has to step in and manually kill something. Each layer exists to protect the layers sitting behind it, and the $47,000 bill is what happens when all four are missing at once.