Model Routing Rules Inside Agent YAML Configurations

Stripe's agents open a lot more pull requests per week now than they did before agent workflows became routine infrastructure there. That's the real story: agents have stopped being a novelty and started being production infrastructure, and at that volume, every decision about which model handles which task compounds into real cost and real quality drift. Run a thousand tasks a day instead of ten, and the gap between routing a summarization job to a frontier model versus a cheaper one stops being a rounding error and starts showing up on the invoice. Routing belongs in a YAML file, not buried in a conditional somewhere in application code, and teams that put it anywhere else lose track of which model handles what within a few months. That's not a stylistic preference. It's the difference between a decision you can review and one you have to go hunting for.

What model routing actually means in an agent context

Model routing is a rule that maps an incoming task to a specific model, evaluated before inference runs. That's the whole definition. But three distinct concerns get lumped together under the same word, and pulling them apart matters more than it sounds like it should.

Task-based routing sends different task types to different models: vision work goes one place, reasoning-heavy work another, summarization somewhere cheaper. Complexity-based routing keeps the task type fixed but varies the model by estimated difficulty, so a simple bug fix and a gnarly refactor don't land on the same endpoint. Fallback routing swaps in a different model for the same task when the primary is down or the budget's run dry.

NeuralTrust's taxonomy splits the mechanics into three families: classifier-based (predict complexity before the call), cascade (run the cheap model first, escalate only if confidence is low), and semantic (embed the query, match it to a domain cluster, route from there). Each carries its own latency and cost profile. A Google Cloud engineering post from August 2025 names the tension outright as the "Router Latency Paradox": a piece meant to make the whole app faster has to be nearly instant itself, or it just becomes a new bottleneck sitting upstream of the one it's supposed to fix. Semantic routing sidesteps this by using vector math instead of a generative call, which resolves in milliseconds rather than the hundreds of milliseconds a second LLM call would cost.

The cost spread across providers isn't subtle, and it's the whole reason routing exists as a discipline rather than an afterthought. Per NeuralTrust, frontier-model tokens run around $2.50 per million input tokens against $0.15 for a smaller variant, a 16x gap. One provider's flash-class model comes in at $0.075, making it 33x cheaper than the frontier option for tasks that don't need frontier-level reasoning. Routing is the policy that decides which of those price points a given task pays for. The model itself just sits behind that policy as a resource, and treating the two as the same thing is exactly how routing logic ends up scattered across a codebase instead of living in one place someone can actually review.

The case for encoding routing rules in YAML rather than application code

Put the routing rule in a YAML file and it becomes something a team can argue about before it ships. It shows up in a pull request diff. Someone can look at a changed line, ask why claude-haiku is suddenly handling a task that used to go to a bigger model, and get an answer before merge instead of after a cost spike lands in finance's inbox.

Version-controlled routing also creates a paper trail nobody has to build by hand: who changed which model handled which task, and when. Teams running agents against production codebases already apply this discipline to infrastructure changes. There's no defensible reason model selection should sit outside it.

There's a testing argument too, and it's not a small one. A YAML schema can be linted, validated in CI, diffed against a prior state, none of which is true of a conditional buried three functions deep. LLMRouter, a routing library out of ulab-uiuc released in December 2025, leans on YAML configuration to control paths and parameters through its data pipeline. That's not an implementation accident, it's a design choice: YAML is the natural surface for saying which model handles what, because a conditional buried in a function hides that decision from the people who most need to see it.

The same logic applies to any declarative config that governs agent behavior: once rules live in YAML committed to a repo, they get reviewed in pull requests and deployed through CI/CD like any other config. Routing rules can follow the identical lifecycle, and there's no structural reason they shouldn't.

Task-based routing: assigning model slots to task types

Hermes Agent's architecture, documented by fast.io in May 2026, is the clearest public example of what task-slot routing looks like once it's actually built. A primary model handles core conversation, tool calls, and streamed responses. Sitting alongside it are eight auxiliary slots: vision, compression, title_gen, approval, session_search, web_extract, skills_hub, and MCP, each independently configurable. Every slot defaults to provider: auto, which just delegates back to the main model, but any slot can be overridden with its own provider, model, base_url, api_key, and timeout. The config supports over 100 models across providers including OpenAI, Anthropic, OpenRouter, Ollama, vLLM, and regional providers like Kimi, MiniMax, Alibaba DashScope, and Tencent TokenHub.

The economics are plain once laid out side by side. Generating a session title doesn't need a reasoning-heavy model. Image analysis needs vision capability the main model may not even have. Compression, left on the primary model, burns expensive tokens on a task any competent summarization model handles for a fraction of the cost. Slot overrides fix all three without touching the main model's config at all.

agentgateway.dev's LLM-based config shows a different flavor of the same idea: header-matched routing, where a model named claude-haiku on the Anthropic provider matches against an incoming x-org: engineering header, while gpt-4 catches requests routed to OpenAI, all declared within a single llm: block carrying policies, models, and matches as sub-keys. Its configuration also supports content-based routing variants that match against request headers to direct traffic to the appropriate backend.

Name the slot after the task, not the model. That's the design rule worth holding onto here. The model sitting inside "title_gen" is just an implementation detail, and it can change next quarter without anyone touching the rule that calls it.

Fallback chains: expressing failure handling as config, not exception logic

Hermes Agent's fallback setup gives a basic fallback_model: block with its own provider and model fields, plus a fallback_providers list for ordered, sequential fallback when one backup isn't enough. The trigger conditions are specific: quota exhaustion and rate limits (429s) after retries fail, server errors (500, 502, 503) after retries fail, auth failures (401, 403) immediately, 404s immediately, and repeated malformed responses. A transient 429 that comes with a Retry-After header doesn't trigger the fallback ladder at all, it just waits it out.

Fallback here is scoped to a single conversational turn. Each new user message starts fresh against the primary model. Within one turn, fallback fires at most once, and if the fallback also fails, normal error handling takes over instead of cascading through an endless chain of backups. There's also a zero-cost resilience option: a provider: custom entry pointing at a local base_url, so a local model can catch the failure case without another API bill attached.

LiteLLM handles a related but distinct trigger: budget exhaustion. When a key's model_max_budget is hit, the request reroutes to a fallback model instead of just failing with a budget_exceeded error. Running out of money and losing a provider connection funnel through the exact same fallback path. Its router_settings block also exposes allowed_fails, which sets a cooldown threshold, and cooldown_time in seconds, which decides how long a failing model gets benched before the router tries it again. Both sit as plain YAML fields next to the routing strategy itself.

A fallback chain written in config is not the same thing as a try/catch block buried in application code. The gap isn't cosmetic: the YAML version is visible to anyone reviewing the file, enforced the same way every single time, and changeable without a code deploy. Anyone still hiding fallback logic inside application code is choosing to make it invisible on every future audit. Rasa's multi-LLM routing, available from version 3.11.0 onward, runs on LiteLLM under the hood and gets enabled by a single router key inside the model group config in endpoints.yml. Fallback and load-balancing logic can share the exact same declarative surface without needing separate systems.

Cost constraints and routing strategy as first-class YAML fields

RouteLLM, a project out of UC Berkeley and Anyscale published at ICLR 2025, trained a classifier-based router on preference data from LMSYS Chatbot Arena. Its matrix factorization router cut cost by 85% on MT Bench while holding onto 95% of GPT-4's performance, sending only 14% of queries to the expensive model. The routing decision was the cost control. There was no separate budget cap layered on top of it, and that's worth sitting with: the cheapest way to control spend is to never call the expensive model in the first place, not to cap it after the fact.

Stanford's FrugalGPT took the cascade route instead: run the cheap model first, escalate only when confidence looks low. The paper reports cost reductions up to 98% at comparable quality. But confidence checks are unreliable for open-ended generation, because models get confidently wrong answers, and that's a real problem for coding agents producing genuinely novel code rather than picking from a known answer set.

LiteLLM's router_settings block exposes a handful of YAML fields that turn cost into something declared rather than discovered after the fact: routing_strategy (options include simple-shuffle, least-busy, usage-based-routing, latency-based-routing), enable_pre_call_checks (set to true, it checks context-window limits before a call goes out, catching an oversized and expensive request before it fires), and per-session iteration and budget caps that keep an agentic loop from running away with the budget.

Plano's approach to agentic loop pinning, documented around August 2026, tackles a subtler failure mode. When pinning is enabled, the router picks a model once, at the start of a new user turn, and every subsequent request inside that same agentic loop sticks with it. That stops a long coding session from switching models mid-task and losing context continuity, and it also stops a cheap task from quietly escalating to an expensive model halfway through a loop.

Cost-optimizing routing wants to cascade to the cheapest model available every single time. Agentic loop stability wants to pin to one model for the whole session. Those two goals pull in opposite directions, plainly, and the YAML has to say which one wins in which context. Leave it ambiguous, and the runtime decides for you, usually at the worst possible moment.

Complexity routing: letting the task signal which model it needs

Three strategies handle complexity routing, and each shows up differently in config. Classifier-based routing runs a lightweight model that predicts task difficulty before the real call happens, then routes on that prediction. LLMRouter supports over 16 router implementations under this umbrella, including BERT-based classifiers, matrix factorization, SVM, MLP, graph-based routers, and Elo rating systems, all controlled through YAML paths and parameters.

Cascade routing skips the pre-classification step entirely: the cheap model runs first, and only escalates if its own confidence comes back low. LiteLLM's proxy config exposes routing strategy options that can approximate this behavior without a separate classifier.

Semantic routing takes a third path, and it's the one worth trusting least for anything conversational. It embeds the incoming query, matches it against a domain cluster, and sends it to the specialist model for that domain regardless of how hard or easy the task actually is: code queries go to a code-specialist model, medical queries to a clinical model, and so on. Per NeuralTrust, it's fast, resolving in milliseconds because it's vector math rather than a generative call, and it scales cleanly as the number of routes grows. But speed isn't the same thing as fit, and teams that pick semantic routing for its speed alone tend to find that out the hard way once real traffic hits it.

xRouteBench's 2026 evaluation found learned routers beating the strongest fixed-model baseline by 14.6% relatively, across combined quality and cost metrics, a real margin for a layer that adds almost no latency of its own. Semantic routing's weak spot is documented: a Google Cloud engineering post notes its effectiveness depends heavily on how good and how complete the example utterances are for each route, and it struggles with multi-turn, context-heavy queries, which is a meaningful constraint for any agent workflow where queries don't stand alone.

Semantic routing isn't a one-time setup, and treating it that way is the mistake most teams make with it. The example utterances per route are themselves data living inside the config file, and they need review and updating every time task definitions shift, the same as any other versioned artifact. LLMRouter's Data Generation Pipeline supports generating that training data and evaluating router performance against 11 benchmark datasets, and its CLI and Gradio interface let a team test a routing strategy interactively before committing it to config.

Tradeoffs between routing approaches and when each pattern breaks down

Task-slot routing is the easiest pattern to audit, and for most teams starting out, it's the right default, full stop. You know before deployment exactly which model handles vision, exactly which one handles compression. It breaks the moment the task taxonomy itself is wrong: a slot labeled "compression" that actually receives a task requiring real reasoning will silently send that task to a model that can't do the job, and nothing in the config flags the mismatch until the output does.

Classifier-based routing works when the query distribution is predictable and stays that way, which is a bigger condition than it sounds like. RouteLLM's BERT classifier reportedly achieved 45% cost savings on MMLU at comparable quality, a solid number, but the approach needs an upfront training step, degrades once real traffic drifts away from the training distribution, and adds a latency hop before every single inference call goes out.

Cascade routing needs no training at all, and FrugalGPT's results show cost savings ranging from 50 to 98% across benchmarks. The tradeoff shows up on unstructured, open-ended work, where confidence checks are unreliable because models can be confidently wrong. That means the cascade needs a quality judge instead of a raw confidence score, and the escalation path adds latency on top of everything else.

Semantic routing is fast, scales well across many domains, and lives entirely inside a config file with no runtime model call needed to make the decision. Its weakness is the one already named: it leans on the completeness of example utterances per route, and it has a documented soft spot for multi-turn, contextual queries, exactly the kind a long coding session tends to produce.

None of these patterns is a universal answer, and anyone selling one as the answer hasn't run it against a real workload. Task-slot routing suits stable, well-understood task boundaries. Classifier and cascade approaches suit teams willing to trade upfront engineering for measurable cost control. Semantic routing suits domain-heavy workloads where the query itself signals which specialist should handle it. Pick the pattern that matches the actual shape of the workload, and write that choice into a YAML file where it can be reviewed, tested, and changed, without anyone having to go spelunking through application code to find where the decision actually lives.