Work out semantic cache hit rate break-even for LLM apps first

A semantic cache charges you on every request and only pays you back on the ones that match. Before you add one to an agent, find out which number decides whether it saves money, because it isn’t the hit rate.

Quick Summary – TLDR:

  • A semantic cache pays an embedding and vector-search fee on every request, hit or miss, so the break-even hit rate is that lookup cost divided by what a full model call costs, adjusted for wrong answers.
  • The full formula is break-even hit rate = L / (C – f x W), where L is lookup cost, C is the cost of a model call, f is the share of hits that are wrong or stale, and W is what one wrong answer costs you.
  • With a $0.0165 model call and a $5 cost per wrong answer, the false-positive rate has to stay under 0.33%, or about one bad hit in 300, before any hit rate can save money.
  • Prompt caching lowers C, so turning it on first roughly halves the false-positive headroom a semantic cache has to work with.
  • In agents, a 1% wrong-hit rate per step becomes a 7.7% chance of at least one wrong hit across an eight-step task.
  • Run the cache in shadow mode first: about 900 graded hits with zero errors is what you need to show a 0.33% error rate at 95% confidence.

Every request pays for a semantic cache before it saves anything. Each one gets embedded, each one triggers a vector search, and only the ones that match an earlier query come back cheaper. That’s why the semantic cache hit rate break-even for LLM apps is the first number to calculate, and it takes one division plus a correction for wrong answers.

Vendors lead with the upside. Redis pitches its LangCache service with claims of up to 70% lower LLM spend, and the GPT Semantic Cache paper on arXiv reported hit rates in the low-to-high 60s percent on its test query sets, with positive-hit accuracy above 97%. Those are results on benchmark traffic full of repeated, similar questions. Your traffic may look nothing like it. Frankly, a hit rate you haven’t measured on your own logs is a guess with a logo on it.

Start with five numbers. C is what one full model call costs you, input and output tokens together. L is what one lookup costs: the embedding call plus the vector search, paid on every request whether it hits or not. p is the hit rate. f is the share of hits that return something wrong or out of date. W is what one wrong answer costs you, in refunds, support time, retries or churn.

Without the correction, the math is plain. With a cache, the average request costs L + (1 – p) x C. Without one, it costs C. The cache wins when p is larger than L / C. That’s the naive break-even, and for expensive calls it comes out tiny.

Now add the wrong answers. The average request becomes L + (1 – p) x C + p x f x W. Solve for p and you get the number to keep:

break-even hit rate = L / (C – f x W)

Read the denominator slowly. Every hit saves you C, but it also carries an expected damage of f x W. If f x W is bigger than C, the denominator goes negative and no hit rate, not even 100%, makes the cache pay.

One more line covers fixed costs. If the vector store, the extra service and the engineering time to maintain them come to F dollars a month, and you handle N requests, then the cache has to clear N x (p x (C – f x W) – L) > F. Small apps usually fail on this line before they fail on any other.

Plugging in real prices

Use assumed prices here and swap in your own. Say a support agent call sends 3,000 input tokens and returns 500. At $3 per million input tokens and $15 per million output tokens, that’s $0.009 plus $0.0075, so C is $0.0165.

The lookup side is cheap. OpenAI lists text-embedding-3-small at $0.02 per million tokens, so embedding a 300-token query costs $0.000006. Add a hosted vector search at a generous $0.00002 per query and L lands near $0.000026.

The naive break-even is then 0.000026 / 0.0165, about 0.16%. One hit in 600 pays for the lookup layer. This is the number that makes teams confident, and it’s the wrong number to be confident about.

Take a 30% hit rate on 2 million requests a month. That’s 600,000 hits at $0.0165, or $9,900 in avoided calls. The lookup tax across all 2 million requests is about $52. Put fixed costs at $800 a month for a managed vector store and upkeep, and you’re netting roughly $9,000 before anyone checks whether the cached answers were right. On a slide, that’s a win.

Where the savings actually go: wrong hits

Now suppose 2% of those hits return something that doesn’t match what the user asked. That’s not an exotic assumption. At a similarity threshold of 0.90, ‘cancel my subscription’ and ‘cancel my order’ can sit close enough to match, and the model never gets a chance to notice the difference because it never sees the question.

Say each wrong answer costs $5 in a support ticket, a refund or a lost customer. Then f x W is 0.02 x $5, or $0.10, about six times C. The denominator in the formula turns negative. In dollars, 600,000 hits at a 2% error rate is 12,000 wrong answers, which is $60,000 in damage against $9,900 in savings.

The shareable version is this. With C at $0.0165 and W at $5, the false-positive rate has to stay under C / W, or 0.33%. That’s one wrong hit in 300. Above that line, no hit rate saves you money, and the cache is a liability you’re paying monthly to run.

W isn’t always $5. For a docs assistant answering ‘what’s the rate limit on the free plan’, a wrong answer might cost you a few cents of confusion, and a 2% error rate at $0.05 is only $0.001 per hit. Set W per query class instead of one global figure. The classes that pass are product facts, definitions and documentation lookups. The classes that fail are anything touching account state, prices, dates or permissions.

Tenant boundaries belong in this calculation too. If two customers share a cache and one gets a hit seeded by the other’s question, you haven’t just served a wrong answer. You’ve leaked data, and W for that case has no ceiling. Partition the index by tenant before you tune anything else.

Stale answers are false positives with perfect similarity

A cached reply about your refund policy is wrong the day the policy changes, even though the incoming question matches the stored one at 0.99. Staleness belongs in f, and it’s the part of f that no similarity threshold can fix.

Put the things that invalidate an answer into the cache key itself: the prompt version, the model version, and a version stamp on whatever documents the answer was built from. When any of them changes, old entries stop matching. Then add a time-to-live sized to how fast the underlying facts move, days for pricing pages, weeks for documentation, and never for anything personalised. Teams that skip the versioning find out about it from a customer, which is the most expensive way to run a test.

Semantic caching vs prompt caching changes your baseline

The two get lumped together, but they do different jobs. Prompt caching reuses the already-processed prefix of your prompt, so the model doesn’t re-read your system prompt and tool definitions at full price. The model still writes a fresh answer every time. Anthropic bills cache reads at 10% of the base input price, and OpenAI discounts cached input tokens as well. Because a new answer gets generated, f is zero by design.

A semantic cache skips the model altogether. That lets it save the output tokens, which prompt caching can’t touch, and it’s the only one of the two that can serve a wrong answer.

The interaction matters for your math. If most of those 3,000 input tokens are a stable system prompt and tool schema, prompt caching cuts the input side from $0.009 to somewhere near $0.0009. C drops from $0.0165 to about $0.0084. Run the false-positive limit again with W at $5 and it falls from 0.33% to about 0.17%. Switching on prompt caching first nearly halves the room a semantic cache has to be wrong in.

So compute C after the discounts you already have, not from the list price. Then notice what’s left: output tokens are now most of the bill, and those are exactly what a semantic cache avoids. That’s the honest case for adding one. It only holds when the answers are safe to reuse.

Why agents make the numbers worse

In a single-turn chatbot, a wrong hit gives one bad reply. In an agent loop, a wrong cached plan or tool choice becomes the input to the next step, and the error carries forward. If each of eight steps has a 1% chance of a wrong hit, the chance that a task contains at least one is 1 – 0.99^8, which is 7.7%. Hold the whole task to the 0.33% budget from earlier and each step has to be wrong no more than about 0.04% of the time.

The lookup bill grows too, since L is paid at every cached step, but at fractions of a cent it’s the smaller problem. The bigger problem is what you embed. The same user sentence can mean different things depending on the tool results already in the conversation, and keying on the last user message alone is the most common design mistake in agent caches. Key on the normalised intent plus the state that changes the answer.

The safer pattern is to cache sub-tasks rather than turns. Read-only steps with stable inputs are good candidates: classifying a ticket, extracting fields from a fixed document, summarising a page that hasn’t changed. Steps that call a tool with side effects, or that depend on live data, stay uncached.

One cheaper option deserves a try before any vector database. An exact-match cache on normalised input, with whitespace and casing stripped, catches true duplicates, costs almost nothing to run and has an f of essentially zero. Many teams find it captures a good share of what they hoped a semantic layer would.

Measure in shadow mode before you build

Don’t ship the cache to find out your hit rate. Run it in shadow mode for two weeks. Embed every request, query the index, log whether it would have hit and at what similarity score, then call the model as usual and serve the model’s answer. You now have p at every threshold, and for each would-be hit you have the cached answer next to the fresh one.

Grade a sample of those pairs. A model judge can do the first pass and a person should settle the disputes. Sample by similarity bucket, for example 0.85 to 0.90, 0.90 to 0.95 and above 0.95, because error rates differ sharply between them. Compute the break-even for each bucket separately. In most setups only the top bucket has a positive net saving, which means your real threshold sits higher than the default most libraries ship with.

Sample size is where teams cut corners. By the statistical rule of three, if you grade n hits and find zero errors, the 95% upper bound on the error rate is roughly 3/n. To show f is under 0.33% you need about 900 clean samples. Grade 100 and find nothing wrong, and all you’ve shown is that f is probably under 3%, which in the example above still loses money.

If the shadow hit rate comes in below your break-even, or you can’t verify f within your budget, stop there. Traffic that looks like a narrow FAQ with high volume and templated phrasing tends to hit. Open-ended conversation rarely does.

The test itself is cheap. Over two weeks at the 2-million-a-month volume used above, you’d run about a million lookups at roughly $0.000026 each, which is around $26.

Also read: How to Calculate Burn Multiple for Startups Before Series A Investors Do • How to set max iterations for AI agent loop before it drains your budget • How to handle LLM API rate limits in production before retries burn your quota

This article is posted in AI News, check it out for more related stories.

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *