Prefill vs decode: why token one dey lag
Prefill na compute-bound and e control time to first token. Decode na memory-bandwidth bound and e control tokens per second. Measure both separately for your box.
Prefill vs decode, for one paragraph
Prefill vs decode na the main distinction wey explain most latency questions about self-hosted LLM (large language model). Prefill dey read the whole prompt for one pass, and compute na wetin limit am. Decode dey write the answer one token at a time, and memory bandwidth na wetin limit am. Time to first token na prefill number. Tokens per second na decode number.
Both phases dey run for the same GPU (graphics processing unit), with the same weights, inside the same process, so e natural to treat dem as one workload. Dem behave like two different programs wey dey share one device. Separate dem, and long list of confusing results no go remain confusing.
Why prefill compute dey bound?
Prefill dey pass the complete prompt through every layer one time. A 2,000 token prompt dey give every matrix multiply 2,000 rows of work, so GPU dey do plenty arithmetic for every byte of weight wey e load. That ratio, arithmetic per byte moved, na arithmetic intensity, and prefill get high one. The device dey run near im compute limit, while memory bus still get spare capacity.
Prefill dey produce two things: the KV cache (the key and value tensors) for every prompt token, and the first output token. Nothing reach reader until that pass finish. Na why, for idle server, prefill dey take most of the time to first token (TTFT). As NVIDIA's benchmarking guide define am, TTFT also include request queuing and network latency. So for busy server, queue wait fit matter as much as the prompt.
Prefill cost dey grow as prompt length increase. The linear part na the per layer matrix work. The quadratic part na attention, where every token dey attend to every earlier token. E start to matter when context long. So if you double the prompt, the prefill part of TTFT go at least double.
You fit watch this within one minute. Send 200 token prompt go your server, then send 2,000 token prompt. Ask for the same number of output tokens each time. TTFT go rise sharply. The streaming speed after the first token hardly change.
Decode na why memory bandwidth bound?
Decode dey produce one token for each step. To produce that one token, GPU must read every weight for the model from memory, use each weight for a few operations, then discard am. Arithmetic intensity dey close to 1, so compute units spend most of their time waiting.
Decode slow because each token need read the whole model from memory. Memory bus na im set the speed, while compute units dey idle.
So, the ceiling for single-stream decode speed na arithmetic wey you fit calculate on paper. Divide memory bandwidth by the number of bytes wey the weights occupy.
Di data wey dey behind dis chart
[
{
"device": "CPU, dual channel DDR5-5600",
"mem_bandwidth_gb_s": 90,
"decode_ceiling_tok_s": 6
},
{
"device": "NVIDIA A10G",
"mem_bandwidth_gb_s": 600,
"decode_ceiling_tok_s": 38
},
{
"device": "NVIDIA L40S",
"mem_bandwidth_gb_s": 864,
"decode_ceiling_tok_s": 54
},
{
"device": "NVIDIA RTX 4090",
"mem_bandwidth_gb_s": 1008,
"decode_ceiling_tok_s": 63
},
{
"device": "NVIDIA A100 80GB SXM",
"mem_bandwidth_gb_s": 2039,
"decode_ceiling_tok_s": 127
},
{
"device": "NVIDIA H100 SXM",
"mem_bandwidth_gb_s": 3350,
"decode_ceiling_tok_s": 209
}
]The bandwidth column get the published specification figure from each vendor. The ceiling column na that figure divided by 16 GB, wey be the size of an 8 billion parameter model stored with 16 bit precision. Na arithmetic calculation, no be benchmark result. Your measured rate go dey below am. Knowing how far below am e dey useful because e go show whether you need fix your serving stack or your hardware.
Read the 6 rows in order, and the pattern clear. CPU for dual-channel DDR5 fit move about 90 GB/s. That one caps decode near 6 tokens per second for that model. L40S dey near 54. H100 SXM, with published bandwidth of 3350 GB/s, dey near 209.
Na this same reason quantization na the strongest single lever for decode speed. Store the same weights at 8 bits instead of 16, and you halve the weight bytes wey GPU must read for each token. So the ceiling roughly doubles. But the ceiling no be the actual result. KV cache still dey read according to its own size. Some 8 bit formats also add work for each token, such as the dynamic per-token activation quantization wey vLLM document for one of its FP8 schemes. Because of this, the measured gain usually dey below 2x. The lever work for the same reason the ceiling move: you move less memory.
How you fit measure each phase for your own server?
Ollama dey return the split inside the response body. Request a non streaming completion, then read the counters.
curl -s http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Explain memory bandwidth in two sentences.",
"stream": false
}' | jq '{prompt_eval_count, prompt_eval_duration, eval_count, eval_duration}'Use model tag wey you don actually pull, and ollama list go show you the tag. prompt_eval_count and prompt_eval_duration na prefill: the prompt token count and the time wey e take process am. eval_count and eval_duration na decode. Durations dey for nanoseconds, so decode speed na eval_count / eval_duration * 1e9 and prefill speed na prompt_eval_count / prompt_eval_duration * 1e9. Ollama API reference talk say prompt_eval_duration cover only uncached prompt tokens, so use prompt wey server never just see to calculate prefill rate. Expect prefill rate to dey much higher than decode rate for the same request. Na this difference everything else here dey explain.
For OpenAI compatible server like vLLM, curl fit measure the first byte time for you.
curl -N -s -o /dev/null \
-w 'pretransfer %{time_pretransfer}s first_byte %{time_starttransfer}s\n' \
http://localhost:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Explain memory bandwidth.", "max_tokens": 128, "stream": true}'time_starttransfer na the time wey the first body byte arrive, so together with "stream": true e include TTFT plus connection setup. Subtract time_pretransfer to remove setup cost. Run am twice and keep the second result, because the first call fit include cold model load.
vLLM still publish the split as Prometheus metrics for /metrics. Run curl -s http://localhost:8000/metrics | grep -E 'time_to_first_token|inter_token_latency' and you go get the histograms vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds. Add vllm:num_requests_running and vllm:num_requests_waiting for queue depth, and vllm:kv_cache_usage_perc for cache pressure. Those five names na the complete dashboard.
When load dey, vllm bench serve --model <name> --num-prompts 200 --request-rate 4 dey drive the running server and report time to first token plus latency for each output token with percentiles. Na the most direct way to see how the two phases dey compete for the stack wey you already dey run. NVIDIA's GenAI-Perf and LLMPerf measure the same quantities if you prefer separate tool. Before you tune anything, take clean baseline: the method for measuring tokens per second for local LLM go give you one wey survive reboot.
Why long system prompt dey delay first token but e no dey delay streaming speed?
Because system prompt na prefill work, and nothing more. Dem process am once, for the same pass with the rest of the prompt, before the first token show. After that pass, e dey exist only as KV cache entries, and decode dey read dem together with everything else. So if prefix caching no dey enabled, system prompt wey get 3,000 tokens go add to TTFT for every single request, while tokens per second go remain almost the same.
Almost, no be exactly. Dem extra KV entries dey read again for every decode step, so very long prompt go slow decode small. The next section explain this.
The fix na to stop recomputing the same prefix. Server wey get prefix caching dey keep the KV cache of shared prefix and reuse am, so the second request wey carry the same system prompt go skip that part of prefill completely. vLLM dey call this automatic prefix caching; check vllm serve --help for your version, because the default don change across releases. That in-GPU KV cache na different thing from the prompt cache wey API provider dey bill you for, and the difference between KV cache and prompt cache worth reading before you tune either one.
Why decode dey slow down as context dey fill?
Two reasons dey, and both concern the KV cache.
The first one na bandwidth. For every decode step, attention reads the keys and values of every token wey come before. Weights get fixed cost per token. KV cache dey grow. You fit calculate the size from the model's config.json: bytes per token equals 2 multiplied by num_hidden_layers, by num_key_value_heads, by the head dimension, by the bytes per element. Use head_dim when the config list am. Use hidden_size divided by num_attention_heads only when e no list am, because some models set the two separately. For example, Mistral NeMo get width of 5,120 and 32 heads, but e get head dimension of 128, no be 160. The leading 2 count one key and one value.
For common 8 billion parameter layout with 32 layers, 8 key and value heads under GQA (grouped query attention), head dimension of 128, and 16 bit precision, na 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. So, 8,000 token conversation carry roughly 1 GB of KV cache for each request.
The second one na capacity. That 1 GB na memory wey no fit hold weights or another user's context. The server set the size of its KV pool once during startup, for vLLM through --gpu-memory-utilization. When the pool full, new requests go wait. vllm:num_requests_waiting wey dey rise while vllm:kv_cache_usage_perc dey near 1 na the exact sign of this condition. Some stacks go preempt request wey dey run and recompute its cache later instead of queueing am. For the user, this go look like stream dey pause halfway.
Long context cost you two times: e need more prefill work for the beginning, and e need more memory read for every token for the rest of the answer.
Why batching dey help throughput but dey make tail latency worse?
Because decode na bandwidth-bound work, extra requests almost no cost anything for compute side. One read of the weights fit produce one token for every sequence wey dey inside the batch, so total throughput dey rise almost linearly as batch size increase. E go continue so until either KV pool finish or batch grow reach point wey compute become the limit again. Continuous batching dey rebuild the batch for every step, so request wey finish go comot and queued request go join without waiting for other requests wey dey inside the batch.
The cost dey show for the percentiles. Each user's next token now dey wait for the slowest part of one shared step. So p50, wey be the median, fit remain acceptable, while p99, wey be the slowest 1 request out of 100, dey stretch. Na p99 people dey notice, because na the pause wey happen for the middle of sentence.
Prefill dey make the matter worse. If one large prompt enter while stream dey continue, e go occupy the device for one long step, and everybody wey dey stream at that time go see a gap. Chunked prefill dey remove most of this by cutting the long prompt into pieces and mixing each piece inside the decode batches. As of October 2026, vLLM V1 engine dey do this by default and exposes the balance through --max-num-batched-tokens. The vLLM tuning documentation explain the tradeoff clearly: smaller values, around 2048, dey give better inter token latency (ITL) because fewer prefills dey interrupt decodes, while larger values dey give better TTFT because more prefill tokens fit enter one batch. That single flag na the prefill-versus-decode balance, exposed as a number wey you fit change. The point where p99 stop being acceptable na capacity question, and how many concurrent users one self-hosted LLM fit serve dey work through am with these same metrics.
Why bigger GPU sometimes no dey change anything?
Because bigger usually mean more compute, and decode no need plenty compute.
Compare two rows for the chart above. A100 80GB get published bandwidth of 2039 GB/s, while L40S get 864 GB/s. The decode ceiling follow exactly: 127 tokens per second against 54. RTX 4090 na very fast card by most measurements, and e 1008 GB/s puts the ceiling at 63. No matter the other differences between two cards, single stream decode dey follow the bandwidth line for the specification sheet.
So, na two ways dey to make decode faster: read fewer bytes per token (quantize the weights, or run smaller model), or buy more bandwidth. Prefill na the opposite case. E need compute, so faster card really fit reduce TTFT for long prompts. If the complaint be say first token dey take four seconds, better hardware fit fix am. If the complaint be say the text dey type slowly, e probably no go fix am.
You suppose run prefill and decode for separate workers?
The big serving stacks dey do exactly this, and dem dey call the technique prefill and decode disaggregation. One pool of workers dey run prefill only, while second pool dey run decode only. The KV cache wey the first pool build dey transfer go the second pool through fast interconnect. E dey work because each phase need different hardware and different scheduling. Prefill need compute and big token batches. Decode need bandwidth and plenty concurrent sequences. When you split dem, each pool fit scale by itself, and one very big prompt no go stall every active stream.
For one VPS (virtual private server) with one GPU, e almost never worth am. You go dey divide one device against itself, and you go turn one pointer into network transfer of gigabytes of cache. The technique become useful when you get enough accelerators to dedicate complete machines to each phase, plus enough steady traffic to keep both pools busy. Below that level, chunked prefill fit provide almost the same isolation with one flag.
Wetin to change when di number no good
When TTFT too high:
- Make di prompt shorter. Prefill cost follow prompt tokens, and if prefix caching no dey on, system prompt cost dey happen for every request.
- Turn on prefix caching so system go calculate repeated prefix once instead of every time.
- Increase
--max-num-batched-tokensso more prefill work fit enter each step. - Check di queue before you blame di model.
vllm:num_requests_waitingwey pass zero mean say di request never start, so na capacity problem.
When tokens per second too low:
- Quantize di weights. Fewer bytes per weight mean fewer bytes to read for each token.
- Check your card published memory bandwidth against di chart above and see how close you dey to di ceiling.
- Reduce
--max-num-batched-tokensso prefill no interrupt decode too often. - Check di context length. If conversation don grow reach thousands of tokens, system go read much bigger KV cache for every step.
Di runtime matter here too, because Ollama and vLLM dey schedule prefill and decode differently, and setting wey help one fit do nothing for di other. Measure first for both phases, then change one thing.
FAQ
Why my first token dey take seconds but the rest dey stream fast?
The wait na prefill, and the streaming na decode. Prefill process the whole prompt for one compute-bound pass before any output dey available, so the cost dey increase as prompt length dey increase. Decode then release one token for each step, at a rate wey memory bandwidth set, and this rate almost no depend on how long the prompt be. Long system prompt for every request na the usual cause. Prefix caching remove the part of the cost wey dey repeat.
Longer prompt dey slow down tokens per second?
Small, and the reason different from TTFT. Every decode step read the keys and values of all previous tokens, so bigger KV cache mean more bytes to read for each token. For common 8 billion parameter layout, the cache na about 128 KiB per token, so 8,000 token context na roughly 1 GB wey system go touch for every step. The bigger effect of long prompt still dey on TTFT, not streaming speed.
Which GPU specification dey predict decode speed?
Memory bandwidth. Divide the published bandwidth by the size of the weights for memory, and you get the arithmetic ceiling for one stream. Card wey get more compute but the same bandwidth no go stream faster. Na also why quantizing weights to 8 bits fit roughly double the decode ceiling: e reduce the weight bytes wey system read for each token by half. The measured gain usually smaller, because system still read the KV cache at its own size, and some 8 bit formats add per-token work such as quantizing activations.
Why throughput dey increase when I add users but each user dey feel slower?
One read of the weights fit serve one token for every sequence inside the batch, so total tokens per second dey rise as batch size dey increase. Each user token now wait for one shared step, so per-user latency dey rise at the same time. Monitor p99 inter-token latency, not the aggregate throughput number, and check vllm:num_requests_waiting to see whether requests dey queue instead of running.