Cutting OpenAI Spend by 82% Using Redis Vector Caching

The exact-match ceiling

Every LLM gateway starts with a hash cache: canonicalize the messages, SHA-256 them, store the completion. It works for byte-identical prompts. But production traffic isn't byte-identical. Users ask "How do I reset my password?" and "How can I reset my password?" and "password reset steps": three hashes, three upstream calls, three bills for the same answer.

Across the fleets we observed before building the semantic tier, exact-match caching captured only 18-24% of repeatable intent. The rest of the repetition was paraphrase traffic that a hash can never see.

Two-tier lookup design

Sentinel's cache pipeline runs in strict order, cheapest check first, upstream last:

• Tier 1: Exact hash (0-1ms). SHA-256 of the canonicalized, PII-scrubbed messages, scoped per tenant and model: cache:{tenant}:{model}:{sha} in Redis.
• Tier 2: Semantic vector (under 50ms). On a Tier 1 miss, the scrubbed prompt is embedded with text-embedding-3-small and compared against the tenant's cached embeddings by cosine similarity.
• Tier 3: Upstream. Only a true miss reaches the provider. The response is then written to both tiers asynchronously.

// Lookup order: exact → semantic → upstream
if hit, ok := cache.GetExact(ctx, tenantID, model, scrubbed); ok {
    return hit // X-Sentinel-Cache-Type: exact
}
vec := embedder.Embed(ctx, scrubbed) // post-scrub text only
if hit, score, ok := semantic.Lookup(ctx, tenantID, model, vec); ok && score >= 0.92 {
    return hit // X-Sentinel-Cache-Type: semantic
}
return router.Route(ctx, req) // full miss → provider

The embedding call adds one small upstream request on Tier 1 misses, but text-embedding-3-small is roughly $0.02 per million tokens, over three orders of magnitude cheaper than the completion calls it eliminates.

Picking the 0.92 threshold

The similarity threshold is the safety dial. Set it too low and you serve stale answers to genuinely different questions; too high and you're back to exact matching. We evaluated thresholds from 0.85 to 0.98 against a labeled set of paraphrase pairs and distinct-but-related questions:

threshold   precision   recall    verdict
0.85        91.2%       84.1%     # too loose, distinct questions collide
0.90        96.8%       76.3%     # acceptable for low-stakes traffic
0.92        99.1%       71.8%     # default, false-hit rate under 1%
0.95        99.7%       58.2%     # conservative, leaves savings on the table

At 0.92, fewer than 1 in 100 semantic hits are false positives, while still capturing over 70% of paraphrase repetition. Team-tier tenants can tune the threshold per workspace. Support bots tolerate looser matching; medical or legal workloads should push toward 0.95.

Tenant isolation & safety

Two invariants are non-negotiable. First, embeddings are generated from post-scrub text: the PII redactor runs before the embedder, so SSNs and secrets never enter the vector index. Second, every cache key and vector is tenant-scoped: there is no cross-tenant similarity path, so one customer's prompts can never serve another customer's traffic. Cache-Control: no-cache or the X-Sentinel-Cache: false header bypasses both tiers for sensitive requests.

Production numbers

Across participating production tenants in the first 30 days:

• 82% reduction in upstream completion calls for high-repetition workloads (support copilots, FAQ agents)
• 24ms median semantic-hit response time, p99 under 50ms, versus 800ms-2s upstream
• $0.00 token cost per cache hit, surfaced per-request in the Command Center's Cache Hits and $ Saved cards