Caching LLM applications

In an LLM application the model call is usually the most expensive thing that happens per request. It costs money per token (the word-pieces a model reads and writes), it takes seconds where a database read takes milliseconds, and much of the work is repeated: the same instructions at the top of every prompt, the same questions from different people, the same document read again and again. Caching an LLM application means finding those repeats and not paying for them twice.

There isn’t one LLM cache. There are four, at different depths, and they save different things:

LayerKeyed onA hit skips, and so saves
Exact response cacheThe whole request, byte for byteThe model call: all tokens, all latency
Semantic cacheThe meaning of the questionThe model call: all tokens, minus one embedding
Prefix cacheThe identical start of the promptRe-reading that start: input-token cost and time to first token
KV cacheTokens already processed in this generationRecomputing them for every new token: work that would grow at least with the square of the length

The first two sit in your application and decide whether to call the model. The last two sit in the inference engine, the server software that runs the model, and make the call cheaper when it happens.

A request travels from the app through the exact response cache, then the semantic cache, then into the model call, where the prefix cache and the KV cache sit. A hit in the exact cache returns the answer and saves all tokens and all latency; a hit in the semantic cache saves all tokens minus one embedding. Inside the model call a prefix-cache hit skips the prefill and saves input work and time to first token, and the KV cache reuses each token's keys and values instead of recomputing them. The first two decide whether to call the model; the last two make the call cheaper. Four caches on one request hit → answer · miss → next layer application · whether to call the model inference engine · makes the call cheaper app request exact response cache key = whole request semantic cache nearest neighbour ≥ τ model call prefix cache shared prompt start KV cache keys · values per token hit · skips the prefill saves input work, time to first token reuse per token no recompute of earlier tokens miss miss hit saves all tokens, all latency hit all tokens minus one embedding answer · generated answer → app
One request, four caches. The first two sit in the application and can return the answer without a model call; the last two sit inside the model call and only make it cheaper.
A request travels from the app through the exact response cache, then the semantic cache, then into the model call, where the prefix cache and the KV cache sit. A hit in the exact cache returns the answer and saves all tokens and all latency; a hit in the semantic cache saves all tokens minus one embedding. Inside the model call a prefix-cache hit skips the prefill and saves input work and time to first token, and the KV cache reuses each token's keys and values instead of recomputing them. The first two decide whether to call the model; the last two make the call cheaper. Four caches on one request app request application · whether to call the model exact response cache key = whole request semantic cache nearest neighbour ≥ τ hit all tokens, all latency hit all tokens minus one embedding engine · makes the call cheaper model call prefix cache shared prompt start KV cache keys · values per token hit · skips prefill input work, time to first token reuse per token no recompute of earlier tokens miss miss answer → app
One request, four caches. The first two sit in the application and can return the answer without a model call; the last two sit inside the model call and only make it cheaper.

Exact response cache

The simplest cache: if this exact request was answered before, return the stored answer. The only hard part is the key, the string the cache looks an answer up by. A response depends on everything that went into the model, so the key has to contain all of it:

Leave one out and the cache returns an answer produced under different conditions. The classic bug is a key built from the user’s message alone: the system prompt changes, and the cache keeps serving answers written under the old one.

Two properties make this layer different from caching an ordinary function.

The same input doesn’t always give the same output. With sampling on, the same prompt gives different answers on different calls, and even at temperature zero most servers don’t promise identical output, because batching and floating-point arithmetic vary from call to call. A cache freezes one sample. Usually that’s a feature: the same question gets the same answer, which is easier to test and to explain. But it means the first answer stored is the one everybody sees, so a bad one is repeated, not just produced once.

Exact matches are rare in free text. “What time do you open on Saturday?” and “what time do you open on saturday” are different strings. Normalising the text (case, whitespace, punctuation) helps a little. Real traffic phrases the same need in dozens of ways, so exact caches work best where the input is generated by a program: a fixed classification prompt run over the same records, a summary of a document that hasn’t changed, a nightly batch job re-run after a crash.

Semantic cache

A semantic cache matches questions by meaning instead of spelling. It uses an embedding model: a small model that turns a piece of text into a list of numbers (a vector, typically a few hundred to a few thousand numbers long) such that texts with similar meaning get vectors pointing in similar directions. How close two vectors are is measured as cosine similarity: 1 for the same direction, 0 for unrelated directions, −1 for opposite. In practice text embeddings rarely use the whole range; unrelated texts often still score well above 0, and where “unrelated” sits depends on the embedding model.

The cache stores pairs of (question vector, answer). For a new question:

  1. Embed the question.
  2. Find the nearest stored question vector.
  3. If its similarity is at or above a threshold τ, return the stored answer. Otherwise call the model, and store the new pair.

“What time do you open on Saturday?”, “are you open saturday morning” and “Saturday opening hours?” land close together, so one model call answers all three. An embedding is far cheaper than a generation, usually by two or more orders of magnitude per request, because the embedding model is small and reads only the question while the generation reads the whole prompt and writes the answer. A vector lookup takes milliseconds, so a hit replaces seconds of generation with almost nothing.

The threshold is the whole design

Everything depends on τ, and it trades two errors against each other:

Two overlapping distributions of cosine-similarity scores for question pairs on an axis from 0 to 1: pairs with the same answer peak high, pairs with different answers peak lower, and the two overlap. The threshold tau is a vertical line. Different-answer pairs scoring above tau are false hits; same-answer pairs scoring below tau are missed hits. Moving tau right trades false hits for misses. The shapes are schematic, not measured. Markers show the article's example pairs: three Saturday opening-hours phrasings as same-answer pairs, and cancel versus uncancel and order 4411 versus 4417 as high-scoring different-answer pairs. Choosing the threshold τ schematic · shapes illustrative, not measured same answer pairs different answer pairs τ ← lower τ · more false hits higher τ · more misses → 0 0.2 0.4 0.6 1 missed hits false hits cosine similarity of a question pair → same answer · Saturday opening hours, three phrasings different answer · cancel / uncancel · order 4411 / 4417
Wherever τ goes, some different-answer pairs score above it (false hits) and some same-answer pairs below it (missed hits). The shapes and the marker positions are illustrative, not measured; the markers stand for the article's example pairs.
Two overlapping distributions of cosine-similarity scores for question pairs on an axis from 0 to 1: pairs with the same answer peak high, pairs with different answers peak lower, and the two overlap. The threshold tau is a vertical line. Different-answer pairs scoring above tau are false hits; same-answer pairs scoring below tau are missed hits. Moving tau right trades false hits for misses. The shapes are schematic, not measured. Markers show the article's example pairs: three Saturday opening-hours phrasings as same-answer pairs, and cancel versus uncancel and order 4411 versus 4417 as high-scoring different-answer pairs. Choosing the threshold τ schematic · not measured same answer different answer τ ← lower τ more false hits higher τ → more misses 0 0.2 0.4 0.6 1 false hits missed hits cosine similarity of a question pair → same answer · Saturday hours ×3 different · cancel/uncancel · 4411/4417
Wherever τ goes, some different-answer pairs score above it (false hits) and some same-answer pairs below it (missed hits). The shapes and the marker positions are illustrative, not measured; the markers stand for the article's example pairs.

A false hit is worse than a miss. A miss costs one model call. A false hit gives a confident, fluent answer to a question nobody asked, and the user can’t tell. So τ is chosen from data, not guessed:

  1. Collect real question pairs and label each one same answer or different answer.
  2. Score every pair with the embedding model.
  3. Pick the lowest τ at which the share of different answer pairs above it is acceptable to you.

Then run the cache in shadow mode first: look up and log what it would have returned, still call the model, and compare. That measures the real hit rate and false-hit rate before any user sees a cached answer.

Making semantic hits safer

Prefix cache

Before a model writes its first word it reads the whole prompt, and that reading (the prefill) costs compute in proportion to the prompt’s length. Most prompts in an application start the same way: the same system instructions, the same tool definitions, the same reference document, the same conversation history from the previous turn. A prefix cache keeps the model’s internal state for a prompt start it has already read, and the next prompt that begins with exactly the same tokens picks up from there.

The rule is strict: the reuse runs from the first token up to the first token that differs, and stops there. Everything after a difference is recomputed, even if the rest is identical. That gives one design rule that matters more than any setting:

Put what never changes first, and what changes last. Stable instructions, then tool definitions, then long reference material, then the conversation, then the new message. A timestamp, a request ID or a user’s name near the top of the prompt makes every prompt unique from that point on and turns the cache off for everything below it.

Token strips of prompts in the recommended order: instructions, tools, reference document, conversation, new message. For the next turn of the same conversation the first differing token is 1,900, so 1,856 tokens are reused after rounding down to a 64-token block and the rest is recomputed. For another conversation sharing the same document the first difference is at token 1,540 and 1,536 tokens are reused. A prompt with a timestamp at token 40 differs there, which rounds down to zero, so nothing is reused and the whole prompt is recomputed. Prefix reuse stops at the first difference stable first, changing last · token counts illustrative instructions tools reference document conversation new message request A computed · state kept next turn same conversation reuse 1,856 of 2,120 another conversation same document reuse 1,536 of 1,790 first differing token · 1,900, rounded down to 1,856 first differing token · 1,540 → 1,536 timestamp near the top next request reuse 0 of 1,910 timestamp first differing token · 40, rounded down to 0 → nothing reused reused recomputed first differing token block boundary · 64-token blocks here
Reuse runs from the first token to the first difference, rounded down to a block, and everything after it is recomputed. A timestamp at token 40 turns the cache off for the whole prompt. Token counts are illustrative.
Token strips of prompts in the recommended order: instructions, tools, reference document, conversation, new message. For the next turn of the same conversation the first differing token is 1,900, so 1,856 tokens are reused after rounding down to a 64-token block and the rest is recomputed. For another conversation sharing the same document the first difference is at token 1,540 and 1,536 tokens are reused. A prompt with a timestamp at token 40 differs there, which rounds down to zero, so nothing is reused and the whole prompt is recomputed. Prefix reuse stops at the first difference stable first, changing last · counts illustrative request A computed · state kept 1 2 3 4 5 next turn, same conversation reuse 1,856 of 2,120 1 2 3 4 5 first difference · 1,900 → 1,856 another conversation, same document reuse 1,536 of 1,790 1 2 3 4 first difference · 1,540 → 1,536 timestamp at token 40 · next request reuse 0 of 1,910 1 2 3 4 5 first difference · 40 → rounds down to 0 reused recomputed first differing token block boundary · 64-token blocks here timestamp 1 instructions · 2 tools · 3 reference document 4 conversation · 5 new message
Reuse runs from the first token to the first difference, rounded down to a block, and everything after it is recomputed. A timestamp at token 40 turns the cache off for the whole prompt. Token counts are illustrative.

What it saves, and what it doesn’t:

A prefix cache doesn’t change what the model computes, only whether it computes it again, so it is safe to use by default. The only design work is the prompt order.

KV cache

Inside every single generation there is one more cache, and without it generation would be impractically slow. A transformer, the architecture behind today’s language models, writes one token at a time, and each new token looks back at every token before it. For each earlier token, each layer of the model computes two vectors, a key and a value, that the new token uses to decide what to pay attention to.

Those keys and values don’t change once computed. The KV cache stores them, so each new token computes its own key and value once and reuses everyone else’s. Without it, writing token number 1,000 would mean recomputing keys and values for the 999 before it, and the total work for an answer would grow at least with the square of its length.

The price is memory. The KV cache grows with every token of the prompt and the answer, for every request running at the same time, and in long-context serving it can take more accelerator memory than the model’s weights. Prefix caching is this same structure kept after the request finishes, so another request can start from it.

Around the model

Two more caches are cheap and often forgotten:

When the cache must forget

An LLM cache has more reasons to go stale than a normal one, because the answer depends on more than the question:

Never cache what must not be repeated: answers built from one user’s private data (unless the cache is partitioned per user), answers about the current time or live state, and anything with side effects. A cached “your payment was sent” is a lie the second time.

Does it pay?

For the two application-side caches the arithmetic is short. Let CL be the cost of a model call, CE the cost of an embedding plus lookup, and h the hit rate. A semantic cache pays the lookup on every request and the model call only on misses:

cost per request = CE + (1 − h) · CL

That beats always calling the model whenever h > CE / CL. With an embedding at a hundredth of a generation, a 1 % hit rate already breaks even, and a 40 % hit rate cuts the cost by about 39 %. Latency works the same way: every request pays tens of milliseconds for the embedding and the lookup, and hits skip seconds.

Cost per request as a fraction of always calling the model, plotted against hit rate h from 0 to 100 percent, from cost equals C E plus one minus h times C L. The no-cache line sits at 1.0. With C E over C L at 0.01 the line starts at 1.01 at zero hits and falls to 0.01 at 100 percent, crossing the no-cache line at the break-even hit rate of 1 percent; at a 40 percent hit rate the cost is 0.61, a 39 percent saving. A second, lighter line for C E over C L at 0.1 breaks even at 10 percent. Does a semantic cache pay? cost per request = CE + (1 − h) · CL · as a fraction of always calling the model 0 0.25 0.5 0.75 1 0.61 cost 0 20 40 60 80 100 hit rate h · % no cache · 1.0 CE / CL = 0.01 CE / CL = 0.1 saved break-even h = CE / CL = 1 % break-even 10 % h = 40 % → 0.61 39 % saved
Cost per request against hit rate, from CE + (1 − h) · CL. With an embedding at a hundredth of a generation the cache pays from a 1 % hit rate; at 40 % it saves 39 %.
Cost per request as a fraction of always calling the model, plotted against hit rate h from 0 to 100 percent, from cost equals C E plus one minus h times C L. The no-cache line sits at 1.0. With C E over C L at 0.01 the line starts at 1.01 at zero hits and falls to 0.01 at 100 percent, crossing the no-cache line at the break-even hit rate of 1 percent; at a 40 percent hit rate the cost is 0.61, a 39 percent saving. A second, lighter line for C E over C L at 0.1 breaks even at 10 percent. Does a semantic cache pay? cost = CE + (1 − h) · CL · fraction of no cache 0 0.5 1 0.61 0 20 40 60 80 100 hit rate h · % no cache · 1.0 CE / CL = 0.01 CE / CL = 0.1 saved break-even · h = 1 % and 10 % h = 40 % → 0.61 39 % saved
Cost per request against hit rate, from CE + (1 − h) · CL. With an embedding at a hundredth of a generation the cache pays from a 1 % hit rate; at 40 % it saves 39 %.

The cost case is almost always made. What decides whether a semantic cache is worth running is the other number: the false-hit rate, which the arithmetic above doesn’t contain, and which only the labelled pairs and the shadow run can tell you.

Things that go wrong

Questions? Let's Chat :)

ai llm engineering