KV cache vs prompt cache explained

KV cache vs prompt cache: the short answer

The KV cache and a provider's prompt cache share a word and almost nothing else. The KV cache is per-request working memory. It sits in your server's RAM or VRAM for the whole life of one request, and it grows with context length and with the number of requests you run at once. Provider prompt caching is a billing and latency feature. A stable prefix of your prompt is stored on the provider's servers, then charged at a discount when you send it again.

One is memory you buy as hardware. The other is memory someone else holds and charges you rent on.

The practical difference matters more than the definition. You can run out of KV cache, and when you do, the model refuses to load or the request is rejected. You cannot run out of prompt cache. You can only fail to hit it, and then you quietly pay full price.

What the KV cache holds, and why it exists

A transformer generating token number 500 has to attend to all 499 tokens before it. For each of those tokens, every layer needs a key vector and a value vector. Recomputing all of them for every new token would make generation grow with the square of the length, so the runtime keeps them instead. That store is the KV cache (key/value cache).

It is per-request state because it is built from that request's exact token sequence. Two users sending different prompts cannot share it, unless the runtime does prefix caching, which is a separate feature described later.

Serving happens in two phases. Prefill reads your whole prompt and fills the cache, and it is limited by compute. Decode produces one token at a time and appends to the cache, and it is limited by memory bandwidth. That split is why prompt processing and token generation report different speeds when you measure tokens per second on your own box.

How much memory does the KV cache use?

Do not go looking for a vendor table. The size is arithmetic you can redo for any model:

bytes per token = 2 * layers * kv_heads * head_dim * bytes per element

The 2 counts the key and the value. Every other number comes from the model's config.json, published on the model's Hugging Face page.

Take Llama 3.1 8B. Its config lists num_hidden_layers 32 and num_key_value_heads 8. The hidden_size of 4096 spread across 32 attention heads gives a head dimension of 128. At f16 each element is 2 bytes:

2 * 32 * 8 * 128 * 2 = 131072 bytes = 128 KiB per token

Multiply that by the context you ask for, then by the requests you run at once.

[
  {
    "label": "2k context",
    "one_request_gib": 0.25,
    "four_requests_gib": 1
  },
  {
    "label": "8k context",
    "one_request_gib": 1,
    "four_requests_gib": 4
  },
  {
    "label": "32k context",
    "one_request_gib": 4,
    "four_requests_gib": 16
  },
  {
    "label": "128k context",
    "one_request_gib": 16,
    "four_requests_gib": 64
  }
]

At 8k of context the cache is 1 GiB. At 32k it is 4 GiB, which is in the same range as the 4-bit weights themselves. At the model's full 128k context it is 16 GiB for one request, and 64 GiB if four requests each fill it. The weights never changed. Only the cache did.

Grouped query attention (GQA) is doing a lot of work in that number. Llama 3.1 8B has 8 key/value heads serving 32 query heads, so four query heads share one stored key/value pair. A model whose num_key_value_heads equals its num_attention_heads uses four times the cache at the same parameter count. Check that one field before you assume two 8B models cost the same to serve.

Why a model that ran at 2k refuses to load at 32k

Because the runtime reserves the KV cache when the model loads, sized for the context length you configured, not for the prompt you actually send. Ollama's default context window is 4096 tokens. Raise it to 32k and you have asked for 4 GiB of extra allocation before a single token arrives.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

The same setting per session, from the interactive prompt:

ollama run llama3.1:8b
/set parameter num_ctx 32768

The failure looks different on each stack. vLLM checks the arithmetic at startup and refuses to run:

ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (78336). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

On a CPU-only VPS there is no such check, because the allocation is ordinary system RAM. The kernel's out-of-memory killer takes the process instead, and it leaves the evidence in the kernel ring buffer:

dmesg -T | grep -i "killed process"

A line naming your serving process means the box promised more memory than it had. The fix is a smaller context, not a bigger swap file: a KV cache paged to disk is read on every single generated token, so generation slows to the point of being useless. Picking a sensible number is covered in our guide to num_ctx and context length in Ollama.

What concurrency does to the number

Every in-flight request carries its own KV cache. That is the line most capacity plans miss. Four users each holding 32k of context need 16 GiB between them, on top of the weights.

Runtimes differ in how strict this is. Ollama and llama.cpp reserve the context you asked for when the model loads, so the memory is committed whether or not anyone uses it. vLLM splits the pool into fixed-size blocks and hands them out as each request grows, so a 500-token request holds only 500 tokens' worth. Either way the pool is finite, and once it is full, new requests queue rather than run. What that queueing does to response times is worked through in how many concurrent users a self-hosted LLM can serve.

Four ways to make the KV cache smaller

1. Lower the context length. This is the biggest lever and usually the cheapest. Most chat workloads never come close to 32k.
2. Quantise the cache itself. Ollama's OLLAMA_KV_CACHE_TYPE defaults to f16 and accepts q8_0, which uses about half the memory, and q4_0, which uses about a quarter. The llama.cpp equivalents are -ctk q8_0 and -ctv q8_0.
3. Pick a model with fewer key/value heads or fewer layers. Read config.json before you download 40 GB of weights.
4. Serve fewer requests at once and queue the rest.

At q4_0 the Llama 3.1 8B figure drops from 128 KiB per token to roughly 32 KiB, so 32k of context costs about 1 GiB instead of 4 GiB. That saving is not free. The keys and values are stored with less precision, so compare output on your own prompts before you keep it.

What provider prompt caching actually buys

Provider prompt caching is a different product with a different unit of account. You mark a stable prefix, the provider stores it, and later calls that repeat the same prefix exactly are billed at a reduced rate instead of full input price.

Anthropic's published multipliers, as of August 2026: a 5-minute cache write costs 1.25 times the base input token price. A 1-hour write costs 2 times, and a cache read costs 0.1 times. Put a 20,000-token system prompt behind those numbers and the shape of the deal becomes obvious.

[
  {
    "label": "No caching, every call",
    "billed_token_equivalents": "20,000"
  },
  {
    "label": "First call, 5 minute cache write",
    "billed_token_equivalents": "25,000"
  },
  {
    "label": "First call, 1 hour cache write",
    "billed_token_equivalents": "40,000"
  },
  {
    "label": "Every later call, cache hit",
    "billed_token_equivalents": "2,000"
  }
]

Read it as arithmetic. The 5-minute write premium is 5,000 token equivalents on the first call: 25,000 against 20,000 for sending it uncached. Every later call inside the window bills 2,000 instead of 20,000, a saving of 18,000. So the 5-minute cache is ahead from the second call onward.

The 1-hour cache is a different bet. It bills 40,000 on the write, a premium of 20,000 token equivalents, so it needs two hits inside the hour before it is ahead. That is a question about your traffic pattern, not about the model. The full calculation, including how to pick the window, is in the break-even math for Claude prompt caching.

Two details decide whether you hit the cache at all. First, a prefix below the model's minimum length is silently not cached: as of August 2026 the documented minimum is 512 tokens for Claude Opus 5 and 1,024 tokens for Claude Sonnet 5, and a shorter request is processed normally with no error returned. Second, the lifetime is measured from the start of the request that writes or reads the entry, and every read refreshes it at no extra cost. A busy endpoint therefore keeps a 5-minute cache alive indefinitely. An endpoint called once every ten minutes pays the write premium every single time and never collects.

Check the response rather than assuming. The usage object reports cache_creation_input_tokens and cache_read_input_tokens. A read count of zero on every call means you are buying writes and getting nothing back.

Where the two caches touch

A long system prompt is the place they meet, and it charges you on both sides at once.

Locally, a 20,000-token system prompt occupies about 2.4 GiB of KV cache on a Llama 3.1 8B server at f16, and it does so separately for every concurrent request that includes it. Remotely, that same prefix costs one cache write and then 0.1 times input on each later call. The local cost scales with your users. The remote cost scales with your traffic and resets during your idle time.

There is a local feature that looks like provider prompt caching and gets confused with it constantly: prefix caching. The vLLM documentation describes automatic prefix caching as caching "the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries". The llama.cpp server keeps a prompt cache per slot by default, and --cache-reuse N sets the smallest chunk it will try to reuse.

What prefix caching saves is prefill compute. Your 20,000-token system prompt is processed once rather than on every request, which cuts time to first token sharply. In vLLM the shared blocks are reused rather than duplicated, so memory improves as well. What it never does is shrink the cache you must hold for the tokens currently live. Keeping the weights resident between requests is a related but separate lever, covered in keeping an Ollama model loaded between requests.

What to measure on your own box

Load the model at your target context, then read real numbers instead of trusting the estimate.

ollama ps
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
free -g

ollama ps lists the loaded model with its size and whether it is running on the GPU or the CPU. A model you expected to sit entirely on the GPU that reports a CPU split means the KV cache pushed part of it out, and generation speed will drop accordingly. nvidia-smi gives the true VRAM figure, and free -g does the same job on a CPU-only VPS. Raise the context in steps, reload, and watch the number move. Your arithmetic and the reported figure should land close to each other. When they do not, the gap is usually the runtime's own compute buffers rather than an error in the formula.

If those numbers push you toward hardware you would rather not rent, the comparison against paying per token is worked out in GPU VPS against API tokens.

FAQ

Is the KV cache the same thing as prompt caching?

No. The KV cache is per-request memory inside the serving process, holding the key and value vectors for every token in the current context. It lives in your RAM or VRAM and is released when the request ends. Provider prompt caching is a billing feature that stores a stable prompt prefix on the provider's infrastructure and charges a reduced rate when you send it again. Running out of KV cache stops a model from loading. Missing the prompt cache only raises your invoice and your time to first token.

Why does my model load at 2k context but fail at 32k?

Because the runtime allocates the whole KV cache at load time, sized for the context length you configured rather than the prompt you send. For Llama 3.1 8B at f16 the cache is 128 KiB per token, so 2k of context costs 0.25 GiB and 32k costs 4 GiB. The weights fit in both cases. The reservation is what fails. vLLM reports it as a ValueError naming the maximum number of tokens it could store, and suggests raising gpu_memory_utilization or lowering max_model_len. On a CPU-only box the kernel out-of-memory killer takes the process instead, which you can confirm with dmesg -T | grep -i "killed process".

How do I calculate KV cache size for my model?

Multiply 2 by the layer count, the number of key/value heads, the head dimension, and the bytes per element. That gives bytes per token. Then multiply by your context length and by the number of concurrent requests. Read the layer and head counts from the model's config.json. Use 2 bytes per element for f16 or bf16. A q8_0 cache is about half that, and q4_0 about a quarter.

Does prompt caching reduce the memory my own server needs?

Provider prompt caching does nothing for your hardware, because the storage sits on the provider's side. The local equivalent is prefix caching, offered by both vLLM and the llama.cpp server. It reuses already-computed key and value vectors for a shared prefix, which saves prefill compute and cuts time to first token. In vLLM the shared blocks are reused rather than duplicated, so memory improves too. Neither feature shrinks the cache required for the tokens currently in flight, so your context and concurrency arithmetic still sets the floor.

Is it worth caching a prompt I only send once?

No. A cache write costs more than plain input, 1.25 times the base rate for the 5-minute option as of August 2026, so a prefix you never resend inside the window is a straight loss. Caching pays when the same prefix repeats, such as a long system prompt or a document you will ask several questions about. Check cache_read_input_tokens in the API response to confirm you are getting hits instead of paying for writes.