Multi-model routing for coding agents

What multi-model routing does to a coding agent

Multi-model routing sends each request to the cheapest model that can handle it. On chat traffic it works well. On a coding agent it usually costs more than it saves, because an agent's bill is dominated by a prompt prefix that is cached per model, and switching models throws that cache away.

The rule this post argues for: route across providers for availability, route across tiers for cost only at task boundaries, and pin one model per session for anything agentic. Everything below is the reasoning.

Four terms, defined once. A router picks a model per request. A gateway is the proxy the request passes through, which may or may not also route. A prompt cache is the provider storing the processed prefix of your prompt, so a later request repeating that prefix is billed at a fraction of the input price. A KV cache (key value cache) is the same idea inside a server you run yourself.

Why chat traffic routes well and agent traffic does not

A chat request is one turn. It arrives, it is classified, it goes to a model, it returns. Nothing carries over to the next one. A router can send this question to a small model and the next to a large one, and neither request knows the other happened. This is the workload almost every routing benchmark measures, and good routers are genuinely good at it.

An agent turn is not one request. One instruction like "fix the failing test" becomes twenty to sixty API calls. Each call re-sends the entire conversation: the system prompt, every tool definition, every file the agent has read, every command output it has seen. The context only grows. By call thirty the repeated prefix can be tens of thousands of tokens, while the genuinely new content in each call is a few hundred.

That shape changes what the word "expensive" means. In chat, cost is roughly the model's price times the request. In an agent loop, cost is the prefix, re-billed on every single call. The rest of this post follows from that one fact.

The prompt cache is per model, and the agent lives inside it

Anthropic prices a cache read at 0.1 times the base input price, and a five minute cache write at 1.25 times. These are the published list prices, as of August 2026.

[
  {
    "label": "Opus 5",
    "uncached_input_usd": "5.00",
    "cache_read_usd": "0.50"
  },
  {
    "label": "Sonnet 5",
    "uncached_input_usd": "2.00",
    "cache_read_usd": "0.20"
  },
  {
    "label": "Haiku 4.5",
    "uncached_input_usd": "1.00",
    "cache_read_usd": "0.10"
  }
]

Read the second series against the first, across rows rather than down them. A cache read on Opus 5 is 0.50 dollars per million tokens. Uncached input on Haiku 4.5, the cheapest model listed, is 1.00 dollars. So re-reading a warm prefix on the most expensive model costs less per input token than reading that same prefix cold on the cheapest one.

That single comparison breaks most routing plans. A router moving work "down" a tier is comparing list prices. But an agent mid-session is not paying list price on the model it is already using. It is paying the cache read price, which already sits below the cheap model's uncached rate.

Caches are keyed on a hash of the prompt prefix, and they are per model. A request to a different model hashes against a store that has never seen it, so it finds nothing and pays full price. The cache is also a hierarchy: tools first, then system, then messages. A change at any level invalidates that level and everything after it, which means editing one tool definition discards the system prompt cache sitting behind it. Agents that register tools at runtime hit this without touching a router at all.

What one mid-session switch actually costs

Take a session with a 40,000 token stable prefix, an ordinary size once an agent has read a handful of files. Below is the prefix cost of a single turn, worked from the list prices above.

[
  {
    "label": "Opus 5, cache warm",
    "prefix_cost_usd": "0.020"
  },
  {
    "label": "Sonnet 5, turn after switch",
    "prefix_cost_usd": "0.100"
  },
  {
    "label": "Opus 5, cache re-warmed",
    "prefix_cost_usd": "0.250"
  }
]

Staying on Opus 5 with a warm cache costs 0.020 dollars for that turn's prefix. The first turn after routing down to Sonnet 5 costs 0.100 dollars, because Sonnet holds no entry for this prefix and has to write one. Coming back to Opus 5 costs 0.250 dollars, because the original entry expired while the session was away.

So the round trip pays two cache writes to avoid two cache reads. Against that, the switch bought one turn of output at Sonnet's output price instead of Opus's. The details block works the whole trip through: the saving lands in fractions of a cent, and the cache penalty lands in tens of cents. The penalty is larger by more than an order of magnitude, and it grows with prefix length while the saving does not.

Opus 5, warm read     40,000 x $0.50 / 1e6  = $0.020
Sonnet 5, cache write 40,000 x $2.50 / 1e6  = $0.100   (1.25 x $2 base)
Opus 5, cache write   40,000 x $6.25 / 1e6  = $0.250   (1.25 x $5 base)

800 x ($25 - $10) / 1e6 = $0.012

Tool call formats are not the same across providers

An agent is a tool-calling loop, so the tool call format matters in a way it never does for chat. Anthropic's Messages API returns a tool_use content block and expects a tool_result block back. OpenAI-compatible APIs return a tool_calls array in which function.arguments is a JSON-encoded string rather than a nested object. A gateway translates between the two, and for ordinary calls the translation is clean.

The problems appear at the edges. Parallel tool calls, where a model emits several calls in one response, are represented differently and are not supported identically everywhere. Strict schema enforcement is a per-provider feature, so a model that guarantees schema-valid arguments on one endpoint only tends toward valid arguments on another. The agent sees the difference as a tool result containing a parse error, which it then tries to repair by spending another turn. Those repair turns are billed at the full prefix price, so a format mismatch shows up on the invoice as well as in the transcript.

Self-hosted endpoints need this configured explicitly. vLLM's OpenAI-compatible server requires --enable-auto-tool-choice together with a --tool-call-parser matched to the model family (hermes, mistral, llama3_json and others), plus a chat template that handles tool-role messages. The vLLM documentation is direct about the limits of this path: with tool_choice="auto" and no strict schema constraint, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema. Picking the wrong parser for your model is a configuration error that presents as an agent that cannot call tools, which is worth knowing before you route traffic at it. The difference between Ollama and vLLM for serving models yourself matters here, because the two expose tool calling on different terms.

A mid-task fallback changes behaviour with no error

Fallback routing is the feature most likely to be enabled by accident. A gateway is configured to retry on another model when the first returns a rate limit or a 5xx, then puts the failed model in a cooldown for some seconds. On chat traffic this is exactly right. Inside a long agent task it means the second half of your task ran on a model you did not choose.

Nothing reports this. The task does not fail, the agent does not warn, and the exit status is success. What you get is a task where the plan was written by one model and the edits were made by another, with a tone and a set of habits that change halfway through. The only reliable signal is the model field in the gateway's request log or the response metadata, so if you run fallbacks at all, log that field per request and read it when a result surprises you. Debugging behaviour without knowing which model produced it wastes more time than the fallback saved.

The same trap catches context compression. Many agents summarise long history by calling a small model. If that call carries a different model or a different system prompt, it writes its own cache entry and does not refresh the main session's, so the next full turn pays a cold prefix. The compression saved tokens and lost the cache.

Routing overhead is real, but latency is not where it hurts

Routers do add work per request, and it is worth being accurate about how much. DigitalOcean reports that their Arch-Router model resolves routing intent in about 51 milliseconds, at 93.17% routing accuracy on their own evaluation. Those are their figures, from their measurement and their benchmark, not ours and not a universal result. Take them at face value and the conclusion is reassuring: 51 milliseconds across forty agent calls is about two seconds added to a task that runs for several minutes.

Two seconds is not what makes routing expensive here. The overhead that does hurt is a router that classifies with a full model call, because that is a second inference on every request, billed and queued like any other. Underneath both sits the cache arithmetic above, which is not overhead at all. It is the cost of the thing routing was supposed to optimise.

On a server you run yourself, the same rule applies with less room to move. The prompt cache's local equivalent is prefix caching in the KV cache, which lives in GPU memory. Hosting two models on one GPU splits that memory between them, so each keeps a smaller KV cache and evicts prefixes sooner. Routing between two local models can therefore reduce the cache hit rate for both at once. If you are sizing hardware for this, the memory and CPU a coding agent actually needs on a VPS is the more useful place to start than a router.

The decision rule

• Route across providers for availability. When the alternative is a failed request, any cost is the right cost. Pin the fallback to a model with the same tool call format so the agent's loop keeps working, and log which model served each call.
• Route across tiers for cost at task boundaries only. Choosing Haiku for a rename and Opus for a refactor is a good decision made once, before the session starts. It is a bad decision made on turn thirty of that session.
• Pin one model per session for anything agentic. A session's value is its warm cache. Treat switching models the way you would treat clearing that cache, because that is what it does.
• Route subagents freely. A subagent that starts with a fresh, small context has no warm cache to lose, so it can run on whatever model suits its job. This is the one place inside an agent where routing is close to free.

For how to build this, the gateway does the work: model aliases and explicit fallback lists. A minimal LiteLLM proxy config looks like this.

model_list:
  - model_name: agent-primary
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: agent-standby
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks: [{"agent-primary": ["agent-standby"]}]
  num_retries: 2
  cooldown_time: 30

Point the agent at agent-primary and it stays on one model until that model is unreachable. Both entries sit on the same provider, so the tool call format does not change when the fallback fires. You still accept a tier change at that moment, which is a trade worth making only because the alternative is a failed request. That is availability routing with no cost routing attached, which is the combination most coding agents want. The full build, including keys and budgets, is covered in running a self-hosted LiteLLM gateway on your own VPS, and this post deliberately does not repeat it.

When one well-chosen model beats any router

Routing is a solution to variance in request difficulty. A coding agent has less of that variance than it looks like, because the expensive part of every call is the same prefix, whatever the call is asking for. Once the prefix dominates, the difference between your cheap tier and your expensive tier shrinks toward the difference in their output prices, and output is a small share of an agent's tokens.

So the honest default is one model, chosen once, with caching turned on and a long enough TTL (time to live) to cover the gaps when you stop to read a diff. Anthropic offers a one hour cache write at 2 times base input, which pays for itself after two reads, and that is often the better lever than any router. Pick the tier deliberately using a straight comparison of Opus, Sonnet and Haiku, and if the bill is still the problem, reduce it with budgets and smaller contexts as in controlling AI agent costs on a VPS rather than with mid-session switching.

Route when requests are independent and short, or when subagents start with fresh contexts. Pin when you have one long session doing one job. Most coding agent work is the second kind, which is why the router that saves money on your chat product will quietly cost you money here. If you have not settled on the agent itself yet, the comparison of Claude Code against Cursor, Codex and Copilot covers how each one handles model selection, and some of them make this decision for you.

FAQ

Does switching models mid-session really lose the prompt cache?

Yes. Prompt caches are keyed on a hash of the prompt prefix and are stored per model, so a request sent to a different model hashes against a store that has never seen that prefix. It finds nothing and pays the full uncached input price, then pays a cache write on top if caching is enabled. Switching back does not recover the original entry either, because the default five minute lifetime has usually expired by then. Check the cache_read_input_tokens and cache_creation_input_tokens fields in the response usage object: a turn that reads zero cached tokens on a long session is the symptom.

Is routing to a cheaper model ever cheaper for an agent?

Only when there is no warm cache to lose. A cache read on Anthropic costs 0.1 times base input, which puts a warm read on Opus 5 below the uncached input rate on Haiku 4.5. Once a session has a large cached prefix, the incumbent model is already the cheap option on input. Routing pays when the context is fresh and small: at the start of a task, or in a subagent that carries only the context it needs.

Why did my agent behave differently halfway through a task?

Check whether a gateway fallback fired. A rate limit or a 5xx on the primary model makes the gateway retry on the standby model and put the primary in cooldown for some seconds, so the rest of the task runs somewhere else. This produces no error and no warning, and the task still reports success. The model field in the gateway request log or the response metadata is the only reliable record, so log it per request if you run fallbacks at all.

Do tool calls work the same across every provider?

Not exactly. Anthropic's Messages API uses tool_use and tool_result content blocks, while OpenAI-compatible APIs use a tool_calls array whose function.arguments is a JSON-encoded string. A gateway translates the common cases well, but parallel tool calls and strict schema enforcement differ per provider. On self-hosted vLLM you must set --enable-auto-tool-choice and a --tool-call-parser matching your model family, and the vLLM documentation notes that without a strict schema constraint the server extracts tool calls from raw text, so arguments may occasionally be malformed.

How long should I set the cache TTL for a coding session?

Use the default five minute lifetime for continuous work, and the one hour option when a human reads diffs between turns. Anthropic prices the five minute write at 1.25 times base input and the one hour write at 2 times, against a read at 0.1 times. The five minute write is repaid by a single read, and the one hour write by two, so on any session where you expect to come back and continue, the longer lifetime usually costs less than paying for a cold prefix.