Cache-Aware Model Routing Is a Long-Horizon Problem

Industry discussion has recently put too much weight on cache affinity. It starts from a reasonable concern: agentic workloads repeatedly send large contexts while generating relatively short responses. A call with 100,000 input tokens and 200 output tokens already reaches a 500:1 ratio. Preserving the cache can make a substantial difference.

But the number that matters is total cost per successfully completed task. A warm cache reduces the cost of repeated context; a better model may eliminate several rounds of work altogether.

In Adaptive LLM Routing with Online RL, we discussed why routing should learn from the outcome of an entire task. Cache affinity fits naturally into that approach. Its value depends on what happens after the next request.

Token Price vs. Task Cost

Recent frontier models make this distinction increasingly relevant. OpenAI reports that GPT-6 Astra improves success while using fewer output tokens on several evaluations. Anthropic reports that Claude Fable 5.1 can match or exceed its predecessor’s performance at lower effort and cost, alongside customer reports of improved token efficiency. (OpenAI, Anthropic)

A model with a higher per-token price can therefore cost less to complete a task. Better decisions, fewer retries, and less unnecessary generation can outweigh the price difference. Conversely, a cheaper model can become expensive when it produces verbose responses or repeatedly follows unproductive paths.

Cache affinity adds another consideration to this calculation. Keeping a struggling model warm makes each attempt cheaper, but every additional attempt still adds cost. Switching to a stronger model can justify both the cold-start cost and the higher token price when it leads to a shorter path to completion.

Cache Affinity Across Turns

An isolated request offers little basis for valuing future cache reuse. A router can account for an existing discount, but evaluating whether to establish or abandon affinity requires looking across requests.

Coding and customer-service workflows already provide that context.

Consider a coding agent whose last three fixes have failed. Its current model has a warm repository cache, while another model would start cold. A successful switch might avoid several more debugging rounds. Similarly, a customer-service conversation may benefit from a stronger model once a routine question develops into a complicated issue.

An RL router observes cache state and cost alongside task progress, model quality, and operational conditions. Across subsequent turns, it learns whether preserving or breaking affinity helped the task succeed. Cache consequences fit into this existing observation and feedback loop, with no separate cache-affinity objective required.

Infrastructure-Level Routing

At the infrastructure layer, cache-aware routing addresses a complementary question: which worker should execute a request for the selected model?

NVIDIA Dynamo and llm-d use information about KV-cache residency and worker load to favor reusable prefixes while accounting for congestion. Better placement reduces redundant prefill computation.

Application-level routing makes the broader choice of which model should handle the next step. Infrastructure routing improves the efficiency of executing that choice. Both contribute to the final task cost.

Learning When to Break Affinity

A cache is valuable because of the work it saves. Its value changes when the task stalls, the conversation shifts, or another model can finish with fewer steps.

An online RL router can learn those patterns from experience, guided by the customer’s objective. A cold cache and a higher token price may be a worthwhile investment when they buy a better path to completion.

Intelligent routing should preserve affinity when it advances the task, abandon it when a better path becomes available, and learn the difference from experience.