Universal Fallbacks: Surviving OpenAI 429 Errors Without User Downtime

What actually fails upstream

Provider outages are rarely total. The failures we see in production are graded: rate-limit 429s during your own traffic spikes, 500/502/503 clusters during provider incidents, and silent network timeouts where the connection simply never answers. A single-provider integration treats all of these as user-facing errors. A gateway should treat them as routing signals.

Retryable error classification

The first job of the failover engine is deciding what deserves a retry on a different provider. Sentinel classifies upstream failures into two buckets:

• Retryable: HTTP 429, 500, 502, 503, 504, and transport timeouts. The request itself is fine; the provider can't serve it right now.
• Terminal: HTTP 400, 401, 403, 404, 422. The request is malformed or unauthorized; replaying it elsewhere would just fail again, slower. These pass through to the caller untouched.

Normalization adapters

Failover across providers only works if the request and response can be translated losslessly. Sentinel normalizes everything to the OpenAI chat-completions schema at the edge, and each adapter handles the provider-specific mapping:

• Anthropic: the system message is lifted to the top-level system parameter of /v1/messages; responses and SSE events are translated back to OpenAI chunk format
• Gemini: messages map to contents: [{role, parts: [{text}]}] with role remapping; generateContent responses normalize back to chat completions
• Groq: OpenAI-compatible, so the adapter is a thin auth-and-endpoint shim using GROQ_API_KEY

Because the caller only ever speaks one schema, a failover from gpt-4o-mini to gemini-2.0-flash is invisible to the client: same request, same response shape, different provider.

The failover state machine

client → sentinel → primary provider
                        ├─ 2xx ────────────────→ stream/return, done
                        ├─ terminal 4xx ───────→ pass through to caller
                        └─ retryable error ────→ fallback chain
                              ├─ equivalent model (e.g. gpt-4o-mini → gemini-2.0-flash)
                              └─ universal safety net: gpt-4o-mini
                                    └─ marks fallback_used=true in trace + usage logs

Two design choices are worth calling out. First, the chain is short and deterministic: one equivalent-model hop, then a universal safety net. Long retry chains multiply tail latency without meaningfully raising success rates. Second, failover works on streams: if the primary provider fails before the first SSE chunk, the fallback stream opens and the client sees uninterrupted tokens. Failures mid-stream are surfaced honestly rather than silently spliced.

Failover observability

Every failover is first-class telemetry, not a log line you'll never find. Traces carry the FALL badge, the Command Center's Fallbacks metric card increments, and the usage log records both the primary and fallback models with their respective status codes, so when you review an incident, you can see exactly which provider failed, when, and what served the traffic instead.