Multi-Model Routing No Dey Always Save for Coding Agents

Wetin multi-model routing dey do to coding agent

Multi-model routing dey send each request go the cheapest model wey fit handle am. E dey work well for chat traffic. But for coding agent, e normally dey cost pass wetin e save, because na prompt prefix wey provider cache per model dey make up most of agent bill, and when you switch model, that cache go lost.

The rule wey this post dey support be say: route across providers for availability, route across tiers for cost only when task change, and pin one model per session for anything wey be agent work. Everything below na the reason.

Four terms, make we define dem once. A router dey choose model for each request. A gateway na proxy wey request dey pass through; e fit also route request, or e fit no route am. A prompt cache na when provider store the processed prefix of your prompt, so later request wey repeat that prefix go cost only small part of the input price. A KV cache (key value cache) na the same idea inside server wey you dey run by yourself.

Why chat traffic dey route well but agent traffic no dey

Chat request na one turn. E go arrive, system classify am, e go go model, then response go return. Nothing carry go the next request. Router fit send this question go small model and send the next one go big model, and neither request know say the other one happen. Na this workload almost every routing benchmark dey measure, and good routers truly dey handle am well.

Agent turn no be one request. One instruction like "fix the failing test" fit become twenty to sixty API calls. Every call go send the complete conversation again: system prompt, every tool definition, every file wey agent don read, and every command output wey e don see. The context dey grow continuously. By call thirty, the repeated prefix fit reach tens of thousands of tokens, while the genuinely new content for each call na only few hundred.

This pattern change wetin the word "expensive" mean. For chat, cost roughly na model price multiply by request. For agent loop, na the prefix dem dey charge again for every single call. Everything for the rest of this post come from this one fact.

The prompt cache na per model, and agent dey live inside am

Anthropic dey charge 0.1 times the base input price for cache read, and 1.25 times for five minute cache write. Na the published list prices be these, as of August 2026.

[
  {
    "label": "Opus 5",
    "uncached_input_usd": "5.00",
    "cache_read_usd": "0.50"
  },
  {
    "label": "Sonnet 5",
    "uncached_input_usd": "2.00",
    "cache_read_usd": "0.20"
  },
  {
    "label": "Haiku 4.5",
    "uncached_input_usd": "1.00",
    "cache_read_usd": "0.10"
  }
]

Read the second series against the first one across rows, no be down the columns. Cache read for Opus 5 na 0.50 dollars per million tokens. Uncached input for Haiku 4.5, wey be the cheapest model for the list, na 1.00 dollars. So, to read warm prefix again for the most expensive model costs less per input token than to read that same prefix cold for the cheapest one.

That one comparison dey break most routing plans. Router wey dey move work “down” one tier dey compare list prices. But agent wey dey inside session no dey pay list price for the model wey e already dey use. E dey pay cache read price, and that price already dey below the uncached rate for the cheap model.

Caches dey use hash of the prompt prefix as key, and dem dey separate per model. Request wey go another model dey hash against store wey never see am before, so e no find anything and e pay full price. Cache too get hierarchy: tools first, then system, then messages. Any change for any level invalidates that level and everything after am. This means say editing one tool definition dey discard the system prompt cache wey dey behind am. Agents wey register tools at runtime fit hit this problem without touching router at all.

Wetin one mid-session switch really dey cost

Take session wey get 40,000-token stable prefix, wey na ordinary size after agent don read some files. Below na the prefix cost for one turn, based on the list prices wey dey above.

[
  {
    "label": "Opus 5, cache warm",
    "prefix_cost_usd": "0.020"
  },
  {
    "label": "Sonnet 5, turn after switch",
    "prefix_cost_usd": "0.100"
  },
  {
    "label": "Opus 5, cache re-warmed",
    "prefix_cost_usd": "0.250"
  }
]

If you remain for Opus 5 with warm cache, e go cost 0.020 dollars for that turn prefix. The first turn after routing go Sonnet 5 go cost 0.100 dollars, because Sonnet no get entry for this prefix and e need write one. If you come back to Opus 5, e go cost 0.250 dollars, because the original entry don expire while the session dey away.

So the round trip pay for two cache writes just to avoid two cache reads. Against this, the switch give one turn output for Sonnet output price instead of Opus price. The details block show the full calculation: the saving na fraction of one cent, while the cache penalty na tens of cents. The penalty pass the saving by more than one order of magnitude, and e dey grow as prefix length grow, while the saving no dey grow like that.

Opus 5, warm read     40,000 x $0.50 / 1e6  = $0.020
Sonnet 5, cache write 40,000 x $2.50 / 1e6  = $0.100   (1.25 x $2 base)
Opus 5, cache write   40,000 x $6.25 / 1e6  = $0.250   (1.25 x $5 base)

800 x ($25 - $10) / 1e6 = $0.012

Tool call format no dey the same for different providers

Agent na tool-calling loop, so tool call format matter for am in a way wey e no dey matter for chat. Anthropic's Messages API dey return tool_use content block and e dey expect tool_result block back. OpenAI-compatible APIs dey return tool_calls array where function.arguments na JSON-encoded string, no be nested object. Gateway dey translate between the two, and for normal calls, the translation clean.

The problems dey show for edge cases. Parallel tool calls, where model dey send several calls for one response, get different representation and no be everywhere dem support am the same way. Strict schema enforcement na feature wey depend on provider, so model wey guarantee schema-valid arguments for one endpoint go only tend toward valid arguments for another one. Agent dey see the difference as tool result wey contain parse error, then e try repair am by using another turn. Dem dey bill those repair turns with full prefix price, so format mismatch go show for invoice and transcript too.

Self-hosted endpoints need make you configure this one clearly. vLLM's OpenAI-compatible server need --enable-auto-tool-choice together with --tool-call-parser wey match the model family (hermes, mistral, llama3_json and others), plus chat template wey fit handle tool-role messages. vLLM documentation talk the limits of this path directly: with tool_choice="auto" and no strict schema constraint, vLLM dey extract tool calls from raw text, so arguments fit sometimes get bad format or violate the function's parameter schema. If you choose wrong parser for your model, na configuration error wey go appear as agent wey no fit call tools. E good make you know this before you route traffic go am. The difference between Ollama and vLLM for serving models yourself matter here, because the two expose tool calling under different conditions.

Fallback wey happen in the middle of task dey change behaviour without error

Fallback routing na the feature wey people most likely enable by mistake. Gateway fit configure to retry with another model when the first one return rate limit or 5xx, then put the failed model for cooldown for some seconds. For chat traffic, this na the correct behaviour. But inside long agent task, e mean say the second half of your task run for model wey you no choose.

Nothing report this. The task no fail, the agent no warn you, and the exit status na success. Wetin you get na task where one model write the plan and another model make the edits, with different tone and habits from the middle of the task. The only reliable signal na the model field for gateway request log or response metadata. So if you dey use fallbacks at all, log that field for every request and check am whenever result surprise you. Debugging behaviour without knowing which model produce am fit waste more time than the fallback save.

The same trap fit affect context compression. Many agents dey summarise long history by calling small model. If that call carry different model or different system prompt, e go write its own cache entry and e no go refresh the main session cache. The next full turn then pay for cold prefix. The compression save tokens but lose the cache.

Routing overhead dey real, but latency no be where e dey hurt

Routers dey add work for every request, and e good make we accurate about the amount. DigitalOcean report say their Arch-Router model dey resolve routing intent for about 51 milliseconds, with 93.17% routing accuracy for their own evaluation. Na their figures, from their measurement and benchmark, no be our own and no be universal result. If you take the figures as dem be, the conclusion dey reassuring: 51 milliseconds across forty agent calls na about two seconds added to task wey dey run for several minutes.

Two seconds no be wetin make routing expensive here. The overhead wey really dey hurt na router wey classify with full model call, because na second inference for every request, and dem bill and queue am like any other one. Underneath both, the cache arithmetic above still dey apply, and e no be overhead at all. Na the cost of the thing wey routing suppose optimise.

For server wey you run by yourself, the same rule still apply, but you get less room to adjust. The local equivalent of prompt cache na prefix caching for KV cache, wey dey live for GPU memory. If you host two models for one GPU, dem go split that memory between dem, so each model go keep smaller KV cache and evict prefixes sooner. Routing between two local models fit therefore reduce cache hit rate for both of dem at the same time. If you dey size hardware for this, the memory and CPU wey coding agent really need for VPS na the more useful place to start than router.

Di decision rule

• Route across providers for availability. If di alternative na failed request, any cost na di correct cost. Pin di fallback to model wey get di same tool call format, so di agent loop go continue to work, and log which model serve each call.
• Route across tiers for cost at task boundaries only. To choose Haiku for rename and Opus for refactor na good decision wey you make once, before di session start. E bad decision if you make am for turn thirty of di session.
• Pin one model per session for anything agentic. Di value of session dey inside di warm cache. Treat model switching like clearing that cache, because na exactly wetin e dey do.
• Route subagents freely. Subagent wey start with fresh, small context no get warm cache to lose, so e fit run on any model wey suit di job. Na only this place inside agent where routing almost free.

For how to build dis one, gateway dey handle di work: model aliases and explicit fallback lists. Minimal LiteLLM proxy config dey look like this.

model_list:
  - model_name: agent-primary
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: agent-standby
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks: [{"agent-primary": ["agent-standby"]}]
  num_retries: 2
  cooldown_time: 30

Point di agent to agent-primary and e go stay on one model until dat model no dey reachable. Both entries dey for di same provider, so di tool call format no go change when fallback fire. You still accept tier change for dat moment, but na trade wey worth am only because di alternative na failed request. Dis na availability routing without cost routing attached, and na di combination wey most coding agents want. Di full build, including keys and budgets, dey covered for how to run self-hosted LiteLLM gateway for your own VPS, and dis post purposely no repeat am.

Wen one model wey you choose well pass any router

Routing na solution for difference wey dey request difficulty. Coding agent get less of this difference than e dey look, because the expensive part of every call na the same prefix, no matter wetin the call dey ask for. Once the prefix become the main cost, difference between your cheap tier and expensive tier go reduce toward difference for their output prices, and output na small part of agent tokens.

So the honest default na one model, choose am once, turn caching on, and use long enough TTL (time to live) to cover the gaps wen you stop read diff. Anthropic dey offer one hour cache write for 2 times base input, and e go pay for itself after two reads. Many times, this na better lever than any router. Choose the tier deliberately with direct comparison of Opus, Sonnet and Haiku, and if na bill still be the problem, reduce am with budgets and smaller contexts as e dey control AI agent costs for VPS instead of switching model for middle of session.

Use routing wen requests dey independent and short, or wen subagents dey start with fresh contexts. Pin the model wen one long session dey do one job. Most coding agent work na the second type, na why the router wey save money for your chat product fit quietly cost you money here. If you never decide on the agent itself, comparison of Claude Code with Cursor, Codex and Copilot explain how each one handle model selection, and some of dem dey make this decision for you.

FAQ

E switch model halfway for session really dey make prompt cache lost?

Yes. Prompt cache dey key on hash of prompt prefix, and dem dey store am per model. So when request go another model, e go hash against store wey never see that prefix before. E no find anything, then e pay full uncached input price. If caching dey enabled, e still pay cache write on top. Switching back no go recover the original entry too, because default five minute lifetime don usually expire by then. Check the cache_read_input_tokens and cache_creation_input_tokens fields inside response usage object. If one long session read zero cached tokens, na the symptom be that.

E ever cheaper to route request go cheaper model for agent?

Na only when no warm cache dey to lose. Cache read for Anthropic cost 0.1 times base input, so warm read on Opus 5 cost less than uncached input rate on Haiku 4.5. Once session get large cached prefix, the model wey dey run already na the cheaper option for input. Routing dey make sense when context fresh and small: for beginning of task, or for subagent wey carry only the context wey e need.

Why my agent come dey behave different halfway through task?

Check whether gateway fallback fire. Rate limit or 5xx for primary model go make gateway retry with standby model and put primary model for cooldown for some seconds. So the rest of task go run somewhere else. This one no produce error or warning, and task still report success. The model field for gateway request log or response metadata na the only reliable record. So log am for every request if you dey use fallbacks at all.

Tool calls dey work the same way for every provider?

No be exactly. Anthropic's Messages API dey use tool_use and tool_result content blocks, while OpenAI-compatible APIs dey use tool_calls array wey function.arguments be JSON-encoded string. Gateway dey translate common cases well, but parallel tool calls and strict schema enforcement dey differ from provider to provider. For self-hosted vLLM, you must set --enable-auto-tool-choice and a --tool-call-parser wey match your model family. vLLM documentation talk say without strict schema constraint, server go extract tool calls from raw text, so arguments fit sometimes malformed.

How long I suppose set cache TTL for coding session?

Use default five minute lifetime for continuous work, and one hour option when human dey read diffs between turns. Anthropic price five minute write at 1.25 times base input and one hour write at 2 times, while read cost 0.1 times. Five minute write pay for itself with one read, and one hour write with two reads. So for any session wey you expect say you go return and continue, longer lifetime usually cost less than paying for cold prefix.