Why We Built Sentinel in Go: Concurrency at 10,000 Requests Per Second

The GIL problem at 10K RPS

Every AI gateway sits on the hot path between your users and the model provider. Each request touches the same pipeline: authenticate the tenant, check quota, scrub PII, probe the cache, pick a provider, stream the response, and write the audit log. None of that is optional, and all of it happens per request, per token stream.

Python-based proxies carry a structural constraint here: the Global Interpreter Lock allows exactly one thread to execute Python bytecode at a time. Async IO helps while you're waiting on the network, but the moment the pipeline does CPU work (regex-based PII scanning, JSON normalization across provider schemas, embedding similarity math), the event loop serializes. Under sustained load, that serialization shows up as tail latency.

The benchmark harness

We benchmarked three configurations fronting the same upstream (gpt-4o-mini, streaming enabled), each behind k6 at a sustained 10,000 requests/sec for 60 seconds on identical 8-core hardware:

• Sentinel: compiled Go binary, full pipeline enabled (PII scrub + exact/semantic cache + failover router)
• Python async proxy: asyncio-based router with an equivalent feature set
• Python sync worker: WSGI-style deployment with a worker pool

We measured added overhead: gateway p99 round-trip minus the upstream provider's own time-to-first-token, so provider jitter doesn't pollute the comparison.

Results: 11ms vs 148ms

# 10,000 RPS sustained, 60s, streaming gpt-4o-mini
sentinel-go        p99_overhead=11ms   errors=0     cpu=61%  rss=214MB
python-async       p99_overhead=148ms  errors=312   cpu=99%  rss=1.9GB
python-sync        p99_overhead=236ms  errors=1,204 cpu=99%  rss=3.4GB

The async Python proxy held up to roughly 3,000 RPS before the event loop started queuing CPU-bound redaction work; past that, p99 climbed near-linearly. The sync worker pool degraded immediately, since every in-flight request pins a worker for the full duration of a stream. Sentinel's overhead stayed flat through 10K RPS with headroom to spare, because CPU work is distributed across OS threads by the Go scheduler rather than funneled through one interpreter lock.

The goroutine fan-out model

Sentinel's request path is a fan-out of cheap goroutines with context.Context cancellation wired end-to-end. A simplified view of the streaming path:

func (h *PromptHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // cancelled the instant the client disconnects

    scrubbed, redactions := h.pii.ScrubMessages(ctx, req.Messages)
    if cached, ok := h.cache.Lookup(ctx, tenantID, req.Model, scrubbed); ok {
        return writeCached(w, cached) // 0-token serve, no goroutines spawned
    }

    stream, err := h.router.RouteStream(ctx, scrubbed) // upstream SSE
    if err != nil { return h.failover(w, r, scrubbed, err) }

    // Each chunk flushes immediately; ctx cancellation tears down
    // the upstream reader without a WaitGroup or callback registry.
    io.Copy(flushWriter{w}, stream)
}

Three properties matter here. First, goroutines are ~2KB stacks, so 10,000 concurrent streams cost tens of megabytes, not gigabytes. Second, the scheduler preempts, so a tenant running a heavy redaction regex can't starve other tenants' streams. Third, cancellation is structural: when a client disconnects, the context tears down the upstream connection, the quota accounting, and the audit write in one signal path.

Memory footprint & deploys

The operational differences compound at deploy time. Sentinel ships as a single static binary: no interpreter, no virtualenv, no dependency resolution in the container boot path. Cold start on Railway is dominated by process exec, not package import. Resident memory under load was 214MB versus 1.9GB for the async Python deployment at the same concurrency, which translates directly into how many gateway replicas you need to pay for per unit of traffic.

Takeaways

• AI gateways do real CPU work in-line. Treat them as compute, not as dumb pipes.
• The GIL doesn't show up in a hello-world benchmark; it shows up at p99 under sustained concurrency with redaction and similarity scoring enabled.
• Compiled concurrency isn't about winning microbenchmarks; it's about flat tail latency while the security pipeline stays on.

If you want to reproduce the harness, spin up a free tenant, point k6 at your gateway key, and watch the trace latencies live in the Command Center. The methodology section of this post is mirrored in our docs.