KV cache vs prompt cache: wetin really dey different?

KV cache versus prompt cache: di short answer

KV cache and provider prompt cache share one word, but almost nothing else. KV cache na per-request working memory. E dey inside your server RAM or VRAM for the whole time one request dey run, and e dey grow as context length increase and as you run more requests at once. Provider prompt caching na billing and latency feature. Stable prefix for your prompt dey stored for provider servers, then dem charge you discounted price when you send am again.

One na memory wey you buy as hardware. The other na memory wey another person dey hold and charge you rent for.

The practical difference matter pass the definition. You fit run out of KV cache, and when that happen, model no go load or dem go reject the request. You no fit run out of prompt cache. You fit only fail to hit am, then you go quietly pay full price.

Wetin KV cache dey hold, and why e dey exist

Transformer wey dey generate token number 500 must attend to all the 499 tokens wey come before am. For each of those tokens, every layer need key vector and value vector. If runtime recompute all of dem for every new token, generation go grow with the square of the length, so runtime keep dem instead. That storage na KV cache (key/value cache).

Na per-request state because e dey build from the exact token sequence for that request. Two users wey send different prompts no fit share am, unless runtime dey use prefix caching. Prefix caching na separate feature wey we go describe later.

Serving dey happen for two phases. Prefill dey read your complete prompt and fill the cache, and compute dey limit am. Decode dey produce one token at a time and append am to the cache, and memory bandwidth dey limit am. Na this split make prompt processing and token generation report different speeds when you measure tokens per second for your own box.

How much memory KV cache dey use?

No go dey find vendor table. The size na arithmetic wey you fit calculate again for any model:

bytes per token = 2 * layers * kv_heads * head_dim * bytes per element

The 2 dey count key and value. Every other number come from the model's config.json, wey dem publish for the model Hugging Face page.

Take Llama 3.1 8B. E config list num_hidden_layers as 32 and num_key_value_heads as 8. The hidden_size of 4096 wey dem spread across 32 attention heads give head dimension of 128. For f16, each element na 2 bytes:

2 * 32 * 8 * 128 * 2 = 131072 bytes = 128 KiB per token

Multiply am by the context wey you request, then by the number of requests wey you run at once.

[
  {
    "label": "2k context",
    "one_request_gib": 0.25,
    "four_requests_gib": 1
  },
  {
    "label": "8k context",
    "one_request_gib": 1,
    "four_requests_gib": 4
  },
  {
    "label": "32k context",
    "one_request_gib": 4,
    "four_requests_gib": 16
  },
  {
    "label": "128k context",
    "one_request_gib": 16,
    "four_requests_gib": 64
  }
]

For 8k context, the cache na 1 GiB for one request. For 32k, e na 4 GiB, wey dey the same range as the 4-bit weights themselves. For the model full 128k context, e na 16 GiB for one request, and 64 GiB if four requests each fill am. The weights no change. Na only the cache change.

Grouped query attention (GQA) dey do plenty work for that number. Llama 3.1 8B get 8 key/value heads wey dey serve 32 query heads, so four query heads share one stored key/value pair. Model wey e num_key_value_heads equal e num_attention_heads dey use four times the cache for the same parameter count. Check that field before you assume say two 8B models go cost the same to serve.

Model wey run for 2k no gree load for 32k

Because runtime dey reserve KV cache when e dey load model. E size am based on the context length wey you configure, no be the prompt wey you actually send. Ollama default no longer be one fixed number. As of October 2026, e dey set context length based on VRAM: 4k for machines wey get under 24 GiB, 32k between 24 and 48 GiB, and 256k for 48 GiB or more. Cloud models dey default to their maximum. CPU-only VPS or small GPU dey start for 4k. If you raise am to 32k, you don request 4 GiB extra allocation before even one token arrive. For 24 GiB card, 32k already be the default, so system go make that reservation whether you request am or not.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

The same setting for each session, from the interactive prompt:

ollama run llama3.1:8b
/set parameter num_ctx 32768

The failure fit look different for each stack. vLLM dey check the calculation when e start and refuse to run:

ValueError: To serve at least one request with the model's max seq len (131072), (16.0 GiB KV cache is needed, which is larger than the available KV cache memory (9.56 GiB). Based on the available memory, the estimated maximum model length is 78336. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.

For CPU-only VPS wey dey run Ollama or llama.cpp, the allocation na normal system RAM, and the shortage fit no show when e start. The kernel out-of-memory killer go terminate the process instead, and e go leave evidence for the kernel ring buffer:

dmesg -T | grep -i "killed process"

If one line name your serving process, e mean say the machine promise more memory than e get. The fix na smaller context, no be bigger swap file: KV cache wey dem page to disk dey read for every generated token, so generation go slow reach the point wey e no useful again. You fit learn how to choose sensible number for our guide about num_ctx and context length for Ollama.

Wetin concurrency do to the number

Every request wey still dey process carry its own KV cache. Na this line most capacity plans dey miss. Four users wey each hold 32k context need 16 GiB between dem, on top of the weights.

Runtimes no handle this matter the same way. Ollama and llama.cpp reserve the context wey you request when model load, so memory dey committed whether anybody use am or not. vLLM divide the pool into fixed-size blocks and give dem out as each request grow, so request wey get 500 tokens hold only memory for 500 tokens. Either way, the pool get limit. Once e full, new requests go queue instead of running. How many concurrent users self-hosted LLM fit serve explain how this queueing affect response times.

Four ways to make KV cache smaller

1. Reduce the context length. Na dis one wey get the biggest effect and e usually cost the least. Most chat workloads no dey come anywhere near 32k.
2. Quantise the cache itself. Ollama's OLLAMA_KV_CACHE_TYPE dey use f16 by default and e accept q8_0, wey dey use about half the memory, plus q4_0, wey dey use about one-quarter. The llama.cpp equivalents na -ctk q8_0 and -ctv q8_0.
3. Choose model wey get fewer key/value heads or fewer layers. Read config.json before you download 40 GB of weights.
4. Serve fewer requests at the same time and queue the remaining ones.

For q4_0, the Llama 3.1 8B figure go drop from 128 KiB per token to roughly 32 KiB, so 32k of context go cost about 1 GiB instead of 4 GiB. This saving no be free. The keys and values dey stored with lower precision, so compare the output with your own prompts before you keep am.

Wetin provider prompt caching really dey give you

Provider prompt caching na different product with different way of counting cost. You mark one stable prefix, provider store am, then later calls wey repeat that same prefix exactly go cost reduced rate instead of full input price.

Anthropic published multipliers, as of October 2026: 5-minute cache write cost 1.25 times the base input token price. 1-hour write cost 2 times, while cache read cost 0.1 times for most models. The newest models read even cheaper: 0.05 times for Claude Opus 5.5 and Sonnet 5.5, and 0.025 times for Claude Fable 5.1 and Mythos 5.1. If you put 20,000-token system prompt behind the standard numbers, the benefit go clear.

[
  {
    "label": "No caching, every call",
    "billed_token_equivalents": "20,000"
  },
  {
    "label": "First call, 5 minute cache write",
    "billed_token_equivalents": "25,000"
  },
  {
    "label": "First call, 1 hour cache write",
    "billed_token_equivalents": "40,000"
  },
  {
    "label": "Every later call, cache hit",
    "billed_token_equivalents": "2,000"
  }
]

Read am as arithmetic. The 5-minute write premium na 5,000 token equivalents for the first call: 25,000 against 20,000 if you send am without cache. Every later call inside the window go bill 2,000 instead of 20,000, so you save 18,000. So the 5-minute cache don pass break-even from the second call onward.

The 1-hour cache na different bet. E bills 40,000 for the write, with premium of 20,000 token equivalents, so e need two hits inside the hour before e pass break-even. This one depend on your traffic pattern, not the model. The full calculation, including how to choose the window, dey for the break-even calculation for Claude prompt caching.

Two details determine whether you go hit the cache at all. First, if prefix shorter than the model's minimum length, system no go cache am silently: as of October 2026, the documented minimum na 512 tokens for Claude Opus 5, Opus 5.5 and Sonnet 5.5, and 1,024 tokens for Claude Sonnet 5. A shorter request go process normally, and system no go return error. Second, lifetime dey count from when the request wey write or read the entry start, and every read refreshes am without extra cost. So busy endpoint fit keep 5-minute cache alive forever. Endpoint wey dem call once every ten minutes go pay write premium every time and never collect any saving.

Check the response instead of assuming. The usage object reports cache_creation_input_tokens and cache_read_input_tokens. If read count na zero for every call, e mean say you dey buy writes but nothing dey come back.

Where local cache and provider cache meet

Long system prompt na where both sides meet, and e dey cost you for both sides at the same time.

For local setup, 20,000-token system prompt dey use about 2.4 GiB of KV cache for Llama 3.1 8B server at f16. E dey use this space separately for every concurrent request wey include am. For provider side, that same prefix go cost one cache write first, then 0.1 times input for each later call, or less for the newest models. Local cost dey increase with your users. Provider cost dey increase with your traffic, and e reset when you no get traffic for some time.

One local feature dey look like provider prompt caching, and people dey often mix dem up: prefix caching. vLLM documentation describe automatic prefix caching as caching "the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries". Current vLLM dey enable am by default. llama.cpp server dey keep prompt cache for each slot by default, and --cache-reuse N sets the smallest chunk wey e go try reuse.

Prefix caching dey save prefill compute. Your 20,000-token system prompt go process once instead of for every request, so time to first token go reduce sharply. For vLLM, shared blocks dey reuse instead of duplicate, so memory usage improve too. But e no ever reduce the cache wey you must hold for tokens wey dey active currently. Keeping the weights resident between requests na related but separate option. keeping an Ollama model loaded between requests cover am.

Wetin to measure for your own box

Load the model with the context size wey you target, then read real numbers instead of trusting the estimate.

ollama ps
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
free -g

ollama ps dey list the loaded model with im size, the context wey e load with, and whether e dey run for GPU or CPU. Check the context column first, because your VRAM now dey determine the default. If model wey you expect to fit entirely for GPU report CPU split, e mean say KV cache push part of am out, and generation speed go reduce. nvidia-smi show the real VRAM figure, while free -g do the same thing for CPU-only VPS. Increase the context small small, reload, and monitor how the number dey change. Your calculation and the reported figure suppose dey close to each other. If dem no close, the difference usually na runtime own compute buffers, not error for the formula.

If these numbers make you consider hardware wey you no really want rent, GPU VPS against API tokens explain the comparison with paying per token.

FAQ

KV cache na the same thing as prompt caching?

No. KV cache na memory for each request inside the serving process. E dey hold key and value vectors for every token for the current context. E dey live for your RAM or VRAM, and e go release when the request end. Provider prompt caching na billing feature. E dey store stable prompt prefix for provider infrastructure, and e charge reduced rate when you send am again. If KV cache finish, model no go load. If prompt cache miss, na your invoice and time to first token go increase.

Why my model dey load for 2k context but e dey fail for 32k?

Because runtime dey allocate the whole KV cache when e load. E size am based on the context length wey you configure, no be the prompt wey you send. For Llama 3.1 8B at f16, cache na 128 KiB per token. So 2k context cost 0.25 GiB, while 32k cost 4 GiB. The weights fit for both cases. Na the reservation dey fail. vLLM report am as ValueError. The message dey show how much KV cache memory e need compared with how much e get. E also show estimated maximum model length, and suggest say make you raise gpu_memory_utilization or lower max_model_len. For CPU-only box wey dey run Ollama or llama.cpp, kernel out-of-memory killer fit kill the process instead. You fit confirm am with dmesg -T | grep -i "killed process".

How I fit calculate KV cache size for my model?

Multiply 2 by the layer count, number of key/value heads, head dimension, and bytes per element. This one give you bytes per token. Then multiply am by your context length and number of concurrent requests. Read the layer and head counts from the model's config.json. Use 2 bytes per element for f16 or bf16. q8_0 cache dey around half of that, while q4_0 dey around one-quarter.

Prompt caching fit reduce the memory wey my own server need?

Provider prompt caching no do anything for your hardware because the storage dey for provider side. The local equivalent na prefix caching. Both vLLM and llama.cpp server support am. E reuse key and value vectors wey dem don already calculate for shared prefix. This one save prefill compute and reduce time to first token. For vLLM, dem reuse shared blocks instead of duplicating dem, so memory usage improve too. But neither feature reduce the cache wey tokens currently in flight need. So your context and concurrency calculation still set the minimum memory requirement.

E make sense to cache prompt wey I go send only once?

No. Cache write cost pass plain input. For the 5-minute option, e cost 1.25 times the base rate as of October 2026. So if you no resend the prefix inside the window, na direct loss. Caching make sense when the same prefix dey repeat, like long system prompt or document wey you go ask several questions about. Check cache_read_input_tokens for the API response to confirm say you dey get cache hits instead of paying for writes.