Beyond 429s: Designing a Zero-Downtime Multi-Provider LLM Fallback Pipeline

The production reality: one provider, one point of failure

If your application hardcodes a single model provider, their incident page is your incident page. Every major upstream (OpenAI, Anthropic, Google, Groq) has logged multi-hour windows of elevated errors in the past year. When your availability target is 99.95%, a single two-hour provider outage consumes your entire monthly error budget, and your status page ends up apologizing for infrastructure you do not control.

The instinctive fix is a try/except around the SDK call with a sleep and a retry. It fails in production for a subtle reason: not all errors deserve a retry. The failures that matter split into two disjoint classes:

• Deterministic errors (HTTP 400, 401, 403, 404, 422): the request itself is wrong. A 401 means the credential is invalid; a 400 means the payload is malformed. Replaying the same bytes against a second provider fails identically, just slower and at double the upstream cost.
• Transient errors (HTTP 429, 500, 502, 503, 504, and network timeouts): the request is fine, the provider cannot serve it right now. These are the only failures where rerouting to a different provider has a chance of succeeding.

A resilient pipeline treats deterministic errors as answers (pass them back to the caller untouched, with the original status code) and transient errors as routing signals. Everything else in this article follows from that classification.

Classify before you retry

The classification rule is small enough to fit on an index card, and it is the highest-leverage function in the entire system. This is the production version, straight from our router:

// fallbackEligible reports whether an upstream failure is transient
// (provider outage, worth retrying on a different provider) or
// deterministic (client/config error, must surface verbatim).
func fallbackEligible(err error) bool {
	var upstream *UpstreamHTTPError
	if errors.As(err, &upstream) {
		// 429 and 5xx: provider-side. 4xx below that: caller's problem.
		return upstream.StatusCode >= 500 || upstream.StatusCode == 429
	}
	// Only network timeouts are transient. A refused or reset
	// connection is deterministic config drift, not an outage.
	var netErr net.Error
	return errors.As(err, &netErr) && netErr.Timeout()
}

One corollary we hold to: cross-provider fallback is an opt-in behavior, gated by an explicit flag, and it only engages for the transient class. If your Groq key is invalid, you should see Groq's 401 in your logs immediately, not a mysterious response from a different model you never asked for.

The blueprint: terminate, scrub, cache, route

Resilience lives in a layer you own, sitting between your client SDKs and the upstream APIs. The data path through that layer looks like this:

client (OpenAI SDK, one base_url change)
   │  POST /v1/chat/completions
   ▼
┌────────────────────────────────────────────────────────┐
│ gateway (reverse proxy, single Go binary)              │
│                                                        │
│  1. auth + quota     key → tenant, atomic reservation  │
│  2. PII scrub        SSN / card / key patterns in RAM  │
│  3. cache lookup     SHA-256 exact, then vector match  │
│  4. router           model prefix → provider adapter   │
└──────┬───────────┬───────────┬───────────┬─────────────┘
       ▼           ▼           ▼           ▼
    OpenAI     Anthropic     Groq       Gemini
    gpt-*      claude-*      llama-*    gemini-*
       │           │           │           │
       └──── transient failure → classify → reroute ─────┘

Two properties of this topology do the heavy lifting. First, the gateway terminates the client connection, so a failover is a server-side reroute: the client's HTTP request stays open, TLS stays intact, and no client code changes when a provider dies. Second, every stage before the router is cheap and deterministic, so the pipeline adds single-digit milliseconds to the p50 and nothing meaningful to the p99.

Why client-side retry loops make outages worse

When a major provider degrades, every client-side retry loop in the world fires at once. A million applications each retrying three times with jittered exponential backoff turns a capacity incident into a self-inflicted DDoS: request volume against the already-damaged provider triples precisely when it can least afford it, error rates climb, more clients trip their retry thresholds, and the feedback loop extends the incident. Engineers call this the thundering herd, and uncoordinated client retries are its favorite food.

Centralizing the retry policy breaks the loop. The gateway applies one classified, budgeted retry per request, routes it to a different provider instead of hammering the degraded one, and can circuit-break a provider entirely once its error rate crosses a threshold. Your fleet's outward behavior during an incident becomes a single controlled policy instead of ten thousand competing backoff implementations.

Failover without schema drift

Cross-provider failover only works if a request written for one model can be served by another without the client noticing. The trick is to normalize everything to a single envelope at the edge (we use the OpenAI chat-completions schema, since every client SDK already speaks it) and let per-provider adapters handle the translation:

• Anthropic: the system message is lifted out of the array into the top-level system parameter of /v1/messages, and max_tokens is always set explicitly because the API requires it. Responses and SSE events are translated back into OpenAI chunk format.
• Gemini: Google's OpenAI-compatible endpoint accepts the envelope directly; the adapter handles auth and the error-envelope differences.
• Groq: fully OpenAI-compatible, so the adapter is a thin endpoint-and-key shim.

Because the client only ever speaks one schema, a failover is invisible at the SDK layer. This is the entire client-side integration, including an explicit per-request fallback directive:

from openai import OpenAI

client = OpenAI(
    base_url="https://sentinelgateway.ai/v1",
    api_key=os.environ["SENTINEL_API_KEY"],
    # One header, per request or client-wide: if the primary
    # model fails transiently, serve from this one instead.
    default_headers={"X-Fallback-Model": "gemini-2.5-flash"},
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize today's deploy log"}],
)
# Same response schema either way. resp.model tells the truth:
# "gpt-4o-mini" normally, "gemini-2.5-flash" if failover engaged.

Honesty in the response envelope is non-negotiable. When a fallback serves the request, the response carries the model that actually produced the tokens in model, the originally requested model in an X-Sentinel-Original-Model header, and a fallback_used marker in the trace. Masking the substitution would quietly corrupt your per-model cost accounting and your eval baselines; surfacing it turns every failover into actionable telemetry.

Cutting latency and wholesale spend with semantic caching

The cheapest fallback is the upstream call you never make. Before the router runs, the gateway checks two cache tiers against the (already PII-scrubbed) prompt:

lookup order, per request:
  1. exact     GET cache:{tenant}:{model}:{sha256(messages)}   → 0 ms
  2. semantic  cosine(prompt_embedding, tenant vector index)   → < 25 ms
               serve when similarity >= 0.92
  3. upstream  route, then async write-back (24 h TTL)

bypass:  Cache-Control: no-cache  → skip reads AND writes
isolation: keys are tenant-scoped; no tenant ever reads
           another tenant's cached answer

The exact tier costs one Redis GET and catches literal repeats. The semantic tier stores a text-embedding-3-small vector per prompt and catches rephrasings ("What is our refund policy?" versus "How do refunds work?"), which is where the hit rate actually comes from: exact matching alone plateaus at low single digits on real chat traffic, while the vector tier pushes production hit rates past 40%. Every hit is served in under 25ms at exactly $0.00 wholesale cost, and during a provider outage the cache absorbs a meaningful fraction of traffic without any failover at all.

Bypass semantics matter as much as hit rate. We never serve cached answers across tenants, never cache safety refusals (a cached denial would outlive the policy fix that caused it), and honor no-cache on both the read and the write path for privacy-sensitive callers. The full implementation deep dive is in our Redis vector caching post.

Actionable takeaways

• Classify before you retry. Transient (429, 5xx, timeouts) reroutes; deterministic (4xx config errors) surfaces verbatim. Never mask a dead API key behind a fallback.
• Keep the chain short. One equivalent-model hop, then one safety net. Long retry chains multiply tail latency without meaningfully raising success rates.
• Normalize at the edge. One canonical envelope, one adapter per provider, honest model fields on every response.
• Cache in front of the router. Exact hash first, vector similarity second, tenant-scoped always. It is the only failover that also cuts your bill.
• Make failover observable. If you cannot graph how often it fires, you do not have resilience; you have a rumor.

Everything above is what SentinelGateway runs in production today, packaged as a single Go binary you can self-host or consume as a managed endpoint. The integration is the Python snippet from earlier: change base_url, keep your SDK, and your existing OpenAI code gains four-provider routing, classified failover, PII scrubbing, and the two-tier cache. The free tier covers 10,000 tokens a month, which is enough to break your staging environment on purpose and watch the fallback fire.