High-Availability Redis for LLM Gateways: Sentinel, Clustering, and Sub-15ms State

Every LLM gateway starts life with in-memory state: a map for rate limits, a small LRU for cached responses. It works until you scale to two replicas, at which point your quota enforcement doubles and your cache hit rate halves. This post covers the move to shared Redis state, why Sentinel is the right HA topology for gateway workloads, how the Python multi-worker pattern used by LiteLLM and most asyncio proxies wires it up, and how a Go gateway keeps quota checks and semantic cache hits under 15ms even while Sentinel is promoting a new master.

In-Memory State Breaks at Replica Two

Two failure modes appear the moment a second gateway replica boots:

• Quota drift. A per-process token bucket allows N requests per replica. Two replicas, double the effective quota. Ten replicas, an order of magnitude over. Your billing math silently detaches from enforcement.
• Cache fragmentation. Each replica warms its own cache. A prompt cached on replica A is a miss on replica B, so your hit rate divides by replica count and your upstream token spend climbs back up.

The fix is shared state in Redis. But a single Redis primary is now your single point of failure: if it dies, every quota check and cache lookup in the fleet fails at once. That is the problem Sentinel solves. Three Sentinel processes watch the primary, and when it stops answering, they vote, promote a replica, and tell clients where the new primary lives.

Sentinel or Cluster

Pick Sentinel when the dataset fits on one node and you need high availability. Pick Cluster when you need sharding because the dataset exceeds one node's memory. Gateway state, meaning quota counters and TTL-bounded cached responses, is measured in low gigabytes for most fleets. Start with Sentinel: one primary, two replicas, three Sentinels. You get automatic failover without resharding logic, and every mature client library speaks the Sentinel protocol.

How Python Proxies Wire Sentinel (the LiteLLM Pattern)

The documented HA path for LiteLLM and similar asyncio proxies runs multiple uvicorn workers per node, each holding its own Sentinel-aware Redis client:

from redis.sentinel import Sentinel

sentinel = Sentinel(
    [("sentinel-1", 26379), ("sentinel-2", 26379), ("sentinel-3", 26379)],
    socket_timeout=0.5,
)
r = sentinel.master_for("llm-gw", password=SENTINEL_PW)

Two costs hide in this shape. First, connection multiplication: 8 workers per node times N nodes, each holding its own pool to the primary plus monitoring connections to all three Sentinels. At 10 nodes you are running hundreds of mostly idle connections, and maxclients becomes a capacity plan. Second, failover discovery is per worker: when Sentinel promotes a new primary, each worker notices independently on its own error path, so the fleet flaps through the promotion window worker by worker instead of once per process.

The Go Path: One Pool, Automatic Re-Pointing

A Go gateway runs one OS process per node, so the topology collapses to a single Sentinel-aware client per replica:

rdb := redis.NewFailoverClient(&redis.FailoverOptions{
    MasterName:     "llm-gw",
    SentinelAddrs:  []string{"sentinel-1:26379", "sentinel-2:26379", "sentinel-3:26379"},
    PoolSize:       128,
    MinIdleConns:   16,
    RouteByLatency: true, // serve cache reads from the nearest replica
})

go-redis subscribes to Sentinel's +switch-master events. When a promotion happens, the client drains the pool and re-points at the new primary once per process, not once per worker. Every goroutine shares the same 128-connection pool, so connection count stays flat as request concurrency grows.

The same client gives you pub/sub for cross-replica invalidation. When one node purges a tenant's cached responses (plan downgrade, abuse flag), it publishes invalidate:{tenant} and every other node drops its local negative-cache entries in the same millisecond.

Staying Atomic Through a Master Failover

The quota check must be one atomic round trip, or concurrent requests race past the limit:

-- KEYS[1] = quota key, ARGV[1] = limit, ARGV[2] = cost, ARGV[3] = TTL seconds
local used = tonumber(redis.call('GET', KEYS[1]) or '0')
if used + tonumber(ARGV[2]) > tonumber(ARGV[1]) then
    return {0, used}
end
local new = redis.call('INCRBY', KEYS[1], ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3], 'NX')
return {1, new}

Lua scripts execute atomically on the node that runs them, which removes the check-then-increment race without any distributed lock. The harder question is what happens during the 5 to 15 seconds while Sentinel is promoting a replica. Three honest properties:

• Redis replication is asynchronous. Writes acknowledged by the old primary in its final seconds may not have reached the promoted replica. For quota counters that means a small overdraft window, not corruption. Size limits with headroom and accept the window; the alternative, blocking on WAIT for replica acks every increment, adds milliseconds to every request to protect against a seconds-long event.
• The demoted primary rejects writes. Clients still pointed at it get READONLY errors. Handle that class explicitly instead of failing the request:

allowed, err := quota.Check(ctx, tenant, cost)
if err != nil {
    if isReadOnly(err) || isConnRefused(err) {
        // Sentinel is mid-promotion: grant a bounded in-memory
        // allowance and reconcile after the pool re-points.
        return grace.Allow(tenant, cost)
    }
    return err
}

• Cache reads and quota writes have different failure budgets. A stale cache read is harmless, so route reads to replicas (RouteByLatency above) and let them serve straight through the failover. A quota write that never lands is lost revenue, so writes go only to the primary and use the bounded grace path during the promotion window.

Measured Latency

From our staging fleet: 3 gateway nodes, one Redis primary plus two replicas, three Sentinels, k6 at 2,000 concurrent streaming requests.

The 15ms cache-hit budget survives because reads never touch the primary. The failover window is real but bounded, and the grace path keeps request latency flat through it.

Build or Drop In

Build this yourself if you want to own Redis topology runbooks. The moving parts are well understood: Sentinel quorum, client re-pointing, quota accounting with a bounded overdraft, replica read routing. None of it is hard individually. The cost is keeping all of it correct while a provider failover and a Redis failover happen at the same time.

SentinelGateway ships this topology as the default: one Go binary, a failover-aware connection pool, Lua-atomic quotas, replica-routed semantic cache, and the bounded grace path during promotions. Integration stays a one-line change:

client = OpenAI(
    base_url="https://gateway.your-domain.com/v1",
    api_key=os.environ["SENTINEL_API_KEY"],
)

Keep your SDK, your prompts, and your provider accounts. The state layer is handled.