Designing a Production LLM Gateway Architecture: Go, Redis Caching, and Sub-25ms Failover

Every team that ships an LLM feature eventually builds the same three things: a retry loop, a cache, and a cost dashboard. Then the first real upstream outage hits, and they discover the retry loop made the outage worse. This post is a walkthrough of the LLM gateway architecture we run in production at SentinelGateway: a single Go binary that sits between your application and the model providers, fails over in under 25ms, caches semantically similar prompts in Redis, and strips PII before anything leaves your perimeter. The patterns apply whether you build your own gateway or drop ours in.

The Operational Reality of LLMs in Production

Client-side retries feel sufficient until your first cascading outage. Three properties of LLM traffic break them.

Retries are correlated. When Anthropic starts returning 529s, every client in your fleet discovers it independently and retries on its own jittered schedule. Thousands of clients backing off on similar curves produce synchronized retry waves that keep the upstream saturated. You become part of the load problem.

Clients cannot share health state. Instance A knows Claude is timing out. Instance B does not, and keeps routing traffic into the dead provider. There is no shared circuit breaker without a central point in the request path.

429s are account-scoped. Rate limit errors apply to your provider account, not to an individual key or process. Retrying the same account accomplishes nothing. The only useful response is failing over to a different provider entirely, which is a routing decision, not a retry decision.

Python-based proxies add a second problem. asyncio runs a single event loop per process, and the GIL pins all JSON parsing, schema translation, and regex work to one core per process. Scaling means forking more uvicorn workers, and each worker duplicates everything: upstream connection pools, Redis connections, in-memory state. Under 2,000 concurrent streaming requests, per-worker GC pauses and loop starvation show up as multi-hundred-millisecond p99 spikes, exactly when you can least afford them: mid-outage, during failover.

The Networking Layer in Go

The gateway's hot path is boring on purpose. A tuned http.Transport with connection pooling sized to provider concurrency:

transport := &http.Transport{
    MaxIdleConns:        1024,
    MaxIdleConnsPerHost: 256, // must exceed per-provider concurrency
    IdleConnTimeout:     90 * time.Second,
    TLSHandshakeTimeout: 5 * time.Second,
    ForceAttemptHTTP2:   true,
}

If MaxIdleConnsPerHost sits below your per-provider concurrency, you pay a full TLS handshake on every overflow request. That alone can add 50 to 100ms under load.

A goroutine per inbound request is cheap: stacks start at 2KB and grow on demand, so two thousand concurrent SSE streams is an unremarkable workload. The streaming path never buffers the full response. It copies event by event and flushes:

flusher := w.(http.Flusher)
scanner := bufio.NewScanner(upstream.Body)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for scanner.Scan() {
    fmt.Fprintf(w, "data: %s\n\n", scanner.Bytes())
    flusher.Flush()
}

Schema translation is where naive implementations burn memory. Do not unmarshal the full request into typed structs and remarshal it for the target provider. Decode only the fields you route on (model, stream, messages) and pass the rest through as json.RawMessage. Combined with sync.Pool byte buffers, the steady-state allocation rate on the hot path stays near zero, which keeps GC out of your p99.

Distributed State with Redis

Quota enforcement has a classic race: check the counter, then increment it. Two concurrent requests both pass the check, both increment, and your quota leaks. The fix is to make check-and-increment a single atomic operation inside Redis, in Lua:

-- KEYS[1] = quota key, ARGV[1] = limit, ARGV[2] = cost, ARGV[3] = TTL seconds
local used = tonumber(redis.call('GET', KEYS[1]) or '0')
if used + tonumber(ARGV[2]) > tonumber(ARGV[1]) then
    return {0, used}
end
local new = redis.call('INCRBY', KEYS[1], ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3], 'NX')
return {1, new}

One round trip, atomic by definition on a single node. Note what is absent: no SET NX PX distributed locks, no lock expiry tuning, no deadlock surface. Lua atomicity replaces the entire category.

Semantic caching follows the same discipline. On the write path, embed the prompt, upsert the vector into a RediSearch HNSW index tagged by tenant and model, and store the response with a TTL. On the read path: embed, run a cosine KNN query, and accept a hit only above a similarity threshold. We run 0.95; below that, false positives cost more than the tokens they save. Pipeline the search and the GET to save a round trip. Measured end to end, cache hits complete in under 15ms, and a hit bills zero tokens.

In-Flight Zero-Retention Data Sanitization

PII redaction happens after authentication and before outbound dispatch, entirely in memory. The prompt that leaves your perimeter is already clean, and the raw form is never written to disk, logs, or Redis.

Two implementation details matter. First, Go's regexp package is RE2: linear-time matching with no catastrophic backtracking, so it is safe to run against adversarial input on the hot path. Compile every pattern once at boot. Second, pure regex produces false positives, so pair it with cheap validators: Luhn checks for card numbers, format checks for cloud API keys, entropy thresholds for bearer tokens.

Streaming responses need care because entities can straddle chunk boundaries. Buffer a sliding window across chunks, redact, then emit. The redaction map lives for the lifetime of the request and is dropped when the handler returns. That is what zero retention means mechanically: there is nothing to subpoena, leak, or forget to delete.

Benchmarks: Goroutines vs Python Async at 2,000 Concurrent Requests

Load rig: k6, 2,000 virtual users, streaming chat completions, upstream latency held at a constant 800ms so the numbers isolate gateway-added overhead.

The medians are close because both runtimes idle well. The tails are not. The Python proxy's p99 spikes come from GC pauses and event loop starvation, and they cluster exactly during failover, when request volume doubles onto the surviving provider. These numbers are from our rig and your mileage will vary, but the shape is stable across runs: goroutine-per-connection I/O holds a flat tail where per-process event loops degrade under correlated load.

Build Your Own or Drop In

Build your own gateway if you have a platform team with spare quarters, compliance requirements that forbid third-party infrastructure in the prompt path, or deep coupling to an internal service mesh. Be honest about the cost: the happy path is a weekend project, but the last 20 percent, streaming edge cases, quota races, provider schema drift, is where the quarters go.

If you want the outcome without owning the pager, SentinelGateway is a single compiled Go binary with no runtime dependencies beyond Redis and Postgres. It is wire-compatible with the OpenAI API, so integration is a one-line change:

client = OpenAI(
    base_url="https://gateway.your-domain.com/v1",
    api_key=os.environ["SENTINEL_API_KEY"],
)

Everything above, failover, semantic caching, PII redaction, quota enforcement, happens below that line. Keep your SDK, your prompts, and your provider accounts. Change where the bytes go.