KV cache vs prompt cache: wetin really dey different?

KV cache versus prompt cache: di short answer

KV cache and provider prompt cache share one word, but almost nothing else. KV cache na per-request working memory. E dey inside your server RAM or VRAM for the whole time one request dey run, and e dey grow as context length increase and as you run more requests at once. Provider prompt caching na billing and latency feature. Stable prefix for your prompt dey stored for provider servers, then dem charge you discounted price when you send am again.

One na memory wey you buy as hardware. The other na memory wey another person dey hold and charge you rent for.

The practical difference matter pass the definition. You fit run out of KV cache, and when that happen, model no go load or dem go reject the request. You no fit run out of prompt cache. You fit only fail to hit am, then you go quietly pay full price.

Wetin KV cache dey hold, and why e dey exist

Transformer wey dey generate token number 500 must attend to all the 499 tokens wey come before am. For each of those tokens, every layer need key vector and value vector. If runtime recompute all of dem for every new token, generation go grow with the square of the length, so runtime keep dem instead. That storage na KV cache (key/value cache).

Na per-request state because e dey build from the exact token sequence for that request. Two users wey send different prompts no fit share am, unless runtime dey use prefix caching. Prefix caching na separate feature wey we go describe later.

Serving dey happen for two phases. Prefill dey read your complete prompt and fill the cache, and compute dey limit am. Decode dey produce one token at a time and append am to the cache, and memory bandwidth dey limit am. Na this split make prompt processing and token generation report different speeds when you measure tokens per second for your own box.

How much memory KV cache dey use?

No go dey find vendor table. The size na arithmetic wey you fit calculate again for any model:

bytes per token = 2 * layers * kv_heads * head_dim * bytes per element

The 2 dey count key and value. Every other number come from the model's config.json, wey dem publish for the model Hugging Face page.

Take Llama 3.1 8B. E config list num_hidden_layers as 32 and num_key_value_heads as 8. The hidden_size of 4096 wey dem spread across 32 attention heads give head dimension of 128. For f16, each element na 2 bytes:

2 * 32 * 8 * 128 * 2 = 131072 bytes = 128 KiB per token

Multiply am by the context wey you request, then by the number of requests wey you run at once.

[
  {
    "label": "2k context",
    "one_request_gib": 0.25,
    "four_requests_gib": 1
  },
  {
    "label": "8k context",
    "one_request_gib": 1,
    "four_requests_gib": 4
  },
  {
    "label": "32k context",
    "one_request_gib": 4,
    "four_requests_gib": 16
  },
  {
    "label": "128k context",
    "one_request_gib": 16,
    "four_requests_gib": 64
  }
]

For 8k context, the cache na 1 GiB for one request. For 32k, e na 4 GiB, wey dey the same range as the 4-bit weights themselves. For the model full 128k context, e na 16 GiB for one request, and 64 GiB if four requests each fill am. The weights no change. Na only the cache change.

Grouped query attention (GQA) dey do plenty work for that number. Llama 3.1 8B get 8 key/value heads wey dey serve 32 query heads, so four query heads share one stored key/value pair. Model wey e num_key_value_heads equal e num_attention_heads dey use four times the cache for the same parameter count. Check that field before you assume say two 8B models go cost the same to serve.

Why model wey run for 2k no gree load for 32k

Runtime dey reserve KV cache when model dey load. E size am based on the context length wey you configure, no be the prompt length wey you actually send. Ollama default context window na 4096 tokens. If you raise am to 32k, you don ask am to allocate 4 GiB extra before even one token arrive.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

You fit use the same setting for each session from the interactive prompt:

ollama run llama3.1:8b
/set parameter num_ctx 32768

The failure fit look different for each stack. vLLM dey check the calculation during startup and refuse to run:

ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (78336). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

For CPU-only VPS, this kind check no dey happen because the allocation na ordinary system RAM. Kernel out-of-memory killer go terminate the process instead, and e go leave evidence for kernel ring buffer:

dmesg -T | grep -i "killed process"

If line mention your serving process, e mean say the box promise more memory than e get. Use smaller context to fix am; no be bigger swap file. KV cache wey page to disk dey read for every generated token, so generation go slow reach point wey e no useful again. Our guide about num_ctx and context length for Ollama explain how to choose sensible number.

Wetin concurrency do to the number

Every request wey still dey process carry its own KV cache. Na this line most capacity plans dey miss. Four users wey each hold 32k context need 16 GiB between dem, on top of the weights.

Runtimes no handle this matter the same way. Ollama and llama.cpp reserve the context wey you request when model load, so memory dey committed whether anybody use am or not. vLLM divide the pool into fixed-size blocks and give dem out as each request grow, so request wey get 500 tokens hold only memory for 500 tokens. Either way, the pool get limit. Once e full, new requests go queue instead of running. How many concurrent users self-hosted LLM fit serve explain how this queueing affect response times.

Four ways to make KV cache smaller

1. Reduce the context length. Na dis one wey get the biggest effect and e usually cost the least. Most chat workloads no dey come anywhere near 32k.
2. Quantise the cache itself. Ollama's OLLAMA_KV_CACHE_TYPE dey use f16 by default and e accept q8_0, wey dey use about half the memory, plus q4_0, wey dey use about one-quarter. The llama.cpp equivalents na -ctk q8_0 and -ctv q8_0.
3. Choose model wey get fewer key/value heads or fewer layers. Read config.json before you download 40 GB of weights.
4. Serve fewer requests at the same time and queue the remaining ones.

For q4_0, the Llama 3.1 8B figure go drop from 128 KiB per token to roughly 32 KiB, so 32k of context go cost about 1 GiB instead of 4 GiB. This saving no be free. The keys and values dey stored with lower precision, so compare the output with your own prompts before you keep am.

Wetin provider prompt caching really dey give you

Provider prompt caching na different product with different way of counting cost. You mark prefix wey no dey change, provider store am, then later calls wey repeat that exact same prefix go cost less instead of full input price.

Anthropic published multipliers, as of August 2026, be these: 5-minute cache write costs 1.25 times the base input token price. 1-hour write costs 2 times, and cache read costs 0.1 times. If you put 20,000-token system prompt behind these figures, the benefit dey clear.

[
  {
    "label": "No caching, every call",
    "billed_token_equivalents": "20,000"
  },
  {
    "label": "First call, 5 minute cache write",
    "billed_token_equivalents": "25,000"
  },
  {
    "label": "First call, 1 hour cache write",
    "billed_token_equivalents": "40,000"
  },
  {
    "label": "Every later call, cache hit",
    "billed_token_equivalents": "2,000"
  }
]

Read am as arithmetic. The 5-minute write premium na 5,000 token equivalents for the first call: 25,000 against 20,000 wey sending am without cache go cost. Every later call inside the window go bill 2,000 instead of 20,000, so you save 18,000. So the 5-minute cache don begin pay from the second call.

The 1-hour cache na different bet. E bills 40,000 for the write, with premium of 20,000 token equivalents, so e need two hits inside the hour before e begin pay. This one depend on your traffic pattern, no be on the model. The complete calculation, including how to choose the window, dey for the break-even calculation for Claude prompt caching.

Two details decide whether you go hit the cache at all. First, if prefix shorter than the model minimum length, e no dey cache and system no go tell you: as of August 2026, the documented minimum na 512 tokens for Claude Opus 5 and 1,024 tokens for Claude Sonnet 5, and shorter request go process normally without error. Second, lifetime dey count from when the request wey write or read the entry start, and every read refreshes am without extra cost. So endpoint wey get plenty calls fit keep 5-minute cache alive forever. Endpoint wey dem call once every ten minutes go pay write premium every time and never collect any benefit.

Check the response instead of assuming. The usage object dey report cache_creation_input_tokens and cache_read_input_tokens. If read count na zero for every call, e mean say you dey buy writes but nothing dey come back.

Wey the two caches dey meet

Long system prompt na where dem dey meet, and e dey charge you for both sides at the same time.

For local side, 20,000-token system prompt dey occupy about 2.4 GiB of KV cache for Llama 3.1 8B server wey dey run f16, and e dey do this separately for every concurrent request wey include am. For remote side, that same prefix cost one cache write, then 0.1 times input for every later call. Local cost dey increase with your users. Remote cost dey increase with your traffic, and e dey reset when you idle.

One local feature dey look like provider prompt caching, and people dey confuse dem all the time: prefix caching. vLLM documentation describe automatic prefix caching as caching "the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries". llama.cpp server keep prompt cache for each slot by default, and --cache-reuse N set the smallest chunk wey e go try reuse.

Prefix caching save prefill compute. Your 20,000-token system prompt process once instead of for every request, so e sharply reduce time to first token. For vLLM, dem reuse the shared blocks instead of duplicating dem, so memory usage improve too. But e never reduce the cache wey you must hold for tokens wey dey active currently. Keeping the weights resident between requests na related but separate option, and keeping an Ollama model loaded between requests cover am.

Wetin to measure for your own box

Load the model with the context wey you target, then read real figures instead of trusting estimate.

ollama ps
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
free -g

ollama ps dey show the loaded model, including e size and whether e dey run for GPU or CPU. If model wey you expect to fit entirely for GPU report CPU split, e mean say KV cache don push part of am comot, and generation speed go reduce accordingly. nvidia-smi dey give the correct VRAM figure, while free -g dey do the same thing for CPU-only VPS. Increase the context small small, reload, and watch how the number dey change. Your calculation and the reported figure suppose dey close to each other. If dem no close, the difference usually na the runtime own compute buffers, not error for the formula.

If these figures make you consider hardware wey you no really want rent, GPU VPS compared with API tokens explain the comparison against paying per token.

FAQ

KV cache and prompt caching dey be the same thing?

No. KV cache na memory for each request inside the serving process. E dey hold key and value vectors for every token inside the current context. E dey your RAM or VRAM, and e go release when the request end. Provider prompt caching na billing feature wey store stable prompt prefix for provider infrastructure. E charge reduced rate when you send the same prefix again. If KV cache finish, model no go load. If prompt cache miss, na your invoice and time to first token go increase.

Why my model dey load for 2k context but e fail for 32k?

Na because runtime dey allocate the complete KV cache when e load. E size am based on the context length wey you configure, no be the prompt wey you send. For Llama 3.1 8B for f16, cache na 128 KiB per token. So 2k context cost 0.25 GiB, while 32k cost 4 GiB. The weights fit for both cases. Na the reservation dey fail. vLLM dey report am as a ValueError wey name the maximum number of tokens wey e fit store. E also suggest make you raise gpu_memory_utilization or lower max_model_len. For CPU-only box, kernel out-of-memory killer go terminate the process instead. You fit confirm am with dmesg -T | grep -i "killed process".

How I fit calculate KV cache size for my model?

Multiply 2 by the layer count, number of key/value heads, head dimension, and bytes per element. This one go give you bytes per token. Then multiply am by your context length and number of concurrent requests. Read the layer and head counts from the model config.json. Use 2 bytes per element for f16 or bf16. A q8_0 cache na about half of that, while q4_0 na about one-quarter.

Prompt caching dey reduce the memory wey my own server need?

Provider prompt caching no dey affect your hardware, because the storage dey for provider side. The local equivalent na prefix caching, and both vLLM and llama.cpp server support am. E reuse key and value vectors wey dem don already calculate for a shared prefix. This one save prefill compute and reduce time to first token. For vLLM, the shared blocks dey reused instead of duplicated, so memory usage improve too. Neither feature reduce the cache wey tokens currently in flight need. So your context and concurrency calculation still set the minimum.

E worth am to cache prompt wey I go send only once?

No. Cache write cost pass plain input. For the 5-minute option, e cost 1.25 times the base rate as of August 2026. So if you no resend the prefix inside the window, na direct loss. Caching dey useful when the same prefix repeat, like long system prompt or document wey you go ask several questions about. Check cache_read_input_tokens for the API response to confirm say you dey get hits instead of paying for writes.