The first LLM feature most teams ship talks to a single model behind a single API key. It works in the demo, it works in the pilot, and then it meets production traffic. A provider has a regional outage. A rate limit trips at 9 a.m. when everyone starts their day. A new model version quietly changes latency. Suddenly the feature that "just called the API" is the feature paging your on-call engineer.
For engineering leaders, the uncomfortable realization is that model reliability is not something you can buy from a provider. Every major provider has downtime, every provider enforces rate limits, and latency varies by model, region, and time of day. The reliability you actually ship is a property of your architecture, not of any one vendor. This piece walks through the routing patterns that make LLM traffic resilient, and the key design principle that lets you adopt them without rewriting application code every time something changes.
Why single-provider architectures fail in production
Three failure modes show up almost universally once usage scales past a pilot:
- Provider outages. Hosted model APIs go down. When your application calls one provider directly and that provider is unavailable, your feature is down for exactly as long as theirs is — and you have no lever to pull. Teams that run critical paths on LLMs increasingly keep a second provider warm precisely so an outage becomes a routing event instead of an incident.
- Rate limits. Providers cap requests and tokens per minute. Under a traffic spike, or when a batch job and interactive traffic collide, you hit those ceilings and requests start returning 429s. A single-provider design has no answer except backing off and hoping.
- Latency variance. The same model can be fast one hour and slow the next depending on provider load and region. If your app is hard-wired to one endpoint, your tail latency is whatever the provider is having that day.
The naive fix — writing retry loops and provider-switching logic into each service — spreads the same brittle code across every application, and every change means a redeploy. That doesn't scale across teams.
The architectural principle: decouple the model name from the model
The pattern that solves this cleanly is to put a routing layer between your applications and the providers, and to have applications call a stable, logical model name rather than a concrete provider endpoint. The application says "give me the production chat model." The routing layer decides, at request time, which real backend that maps to — and can change that mapping without the application knowing.
This is the core idea behind a unified AI gateway: a proxy that exposes one consistent, standardized interface and handles load balancing, health-aware routing, retries, and fallbacks on the applications' behalf. Because the application only ever knows the logical name, you can add a provider, shift weights, or fail traffic over during an incident with a configuration change instead of a code change across a dozen services.
Four routing strategies worth knowing
Once traffic flows through a routing layer, you can choose how it distributes requests across targets. Four strategies cover most needs:
- Weight-based routing. Split traffic across targets by percentage — say 80% to one deployment and 20% to another. This is the workhorse for load distribution and for canarying: send 5% of traffic to a new model, watch the metrics, and ramp only when you trust it.
- Latency-based routing. Automatically send each request to whichever healthy target currently has the lowest latency. Because latency drifts over time and region, this lets the system continuously chase the fastest option rather than betting on a static choice.
- Priority-based routing. Define an ordered preference — primary, secondary, tertiary — and route to the highest-priority healthy target, falling through to the next when one is unavailable. This is the classic active-passive failover pattern.
- Complexity or intent-based routing. Classify each request and route accordingly — simple requests to a cheaper, faster model and hard ones to a more capable model. This blends reliability with cost control.
Retries and fallbacks: the difference matters
Two mechanisms do the heavy lifting during failures, and it's worth being precise about the distinction.
A retry re-attempts the same target — useful for transient errors like a momentary 500 or a brief 503. You configure how many attempts, how long to wait between them, and which status codes should trigger a retry (429s and 5xxs are typical).
A fallback moves the request to a different target when the first one can't serve it. You define which status codes trigger a fallback and which targets are eligible to receive that redirected traffic. Retries handle blips on one backend; fallbacks handle one backend being genuinely unhealthy. A resilient configuration uses both: retry the primary a couple of times for transient errors, then fall back to a secondary provider if it's truly down.
TrueFoundry's virtual models are a concrete example of packaging all of this behind one name: you point applications at a single identifier, then configure a routing strategy and a set of real target models underneath it, with per-target retry and fallback rules. The application code never changes when you adjust the routing — which is exactly the property you want.
A practical rollout
You don't need to boil the ocean. A sensible sequence for a team adopting this:
Start by routing existing traffic through the gateway using the logical-name pattern, even if there's only one backend behind each name. This costs almost nothing and immediately decouples your apps from provider specifics. Next, add a second provider or deployment for your most critical model and configure priority-based fallback, so an outage becomes a routing event. Then layer in retries tuned to the error codes you actually see in your logs. Finally, once you trust the plumbing, use weight-based routing to canary new models safely and latency-based routing to squeeze down tail latency.
What to watch for
A few caveats keep this honest. Not every operation supports routing — asynchronous batch jobs, for instance, typically run against a single backend because there's no live request to fail over. Fallback across providers only works if the target models are genuinely interchangeable for your use case, so validate output quality across the providers you route between, not just their uptime. And routing is only as good as your health signals: invest in the observability that tells the router which targets are actually healthy, and tune your retry and fallback status codes to the failures your traffic really produces.
The takeaway
Reliability for LLM-powered features is an architecture decision, not a vendor guarantee. By putting a routing layer between your applications and your model providers, and having applications call stable logical names, you turn outages, rate limits, and latency spikes into configuration-time problems rather than incidents that require code changes. Weight, latency, and priority strategies, combined with well-tuned retries and fallbacks, give you the levers to keep traffic flowing when any single provider has a bad day. The teams that treat multi-provider routing as foundational infrastructure — rather than something to bolt on after the first outage — are the ones whose AI features stay up.
