What Is LLM Routing? Cost, Quality, Latency, and Residency

The problem routing solves

Most AI-powered applications don't send one type of request — they send many. A support assistant might handle a simple FAQ lookup and a complex multi-step troubleshooting conversation with the same integration. Hardcoding one model for both means either overpaying for the simple case or under-serving the complex one. LLM routing is the practice of sending each request to whichever model actually fits it, rather than picking one model for an entire application and living with the mismatch.

How a routing decision gets made

A router evaluates a request against whatever dimensions matter for that request, then picks a model or provider that satisfies them. The dimensions vary by what a team cares about, but they typically include:

• Cost — provider pricing per token, and whether a cached response could answer the request without new inference at all
• Quality — which model performs best for the specific type of task being asked
• Latency — response time, including network distance to the provider's infrastructure
• Residency — for organizations with data-residency requirements, whether a given provider is even a permitted destination for this specific request

Static routing vs. dynamic routing

Some routing is static — a fixed rule like "coding requests always go to model X." Some is dynamic — the routing decision is made per request, potentially based on the content or classification of that specific request. Static rules are simpler and more predictable; dynamic routing adapts better to workloads where request types genuinely vary. Most real systems end up using a mix: static rules for hard constraints (like residency), dynamic selection within whatever's left.

Where this fits into infrastructure

Routing is typically implemented at a gateway layer — a single integration point that sits between an application and every provider it uses, making the routing decision on every request rather than requiring the application itself to know about every provider. That's the architecture behind how Mizan routes AI requests.