Prefill vs decode: why token one dey lag

Prefill vs decode, for one paragraph

Prefill vs decode na the main distinction wey explain most latency questions about self-hosted LLM (large language model). Prefill dey read the whole prompt for one pass, and compute na wetin limit am. Decode dey write the answer one token at a time, and memory bandwidth na wetin limit am. Time to first token na prefill number. Tokens per second na decode number.

Both phases dey run for the same GPU (graphics processing unit), with the same weights, inside the same process, so e natural to treat dem as one workload. Dem behave like two different programs wey dey share one device. Separate dem, and long list of confusing results no go remain confusing.

Prefill compute-bound dey why?

Prefill dey push the complete prompt pass through every layer once. For 2,000-token prompt, every matrix multiply get 2,000 rows of work, so GPU dey do plenty arithmetic for each byte of weight wey e load. That ratio, arithmetic per byte wey move, na arithmetic intensity, and prefill get high arithmetic intensity. The device dey run near im compute limit, while memory bus still get spare capacity.

Prefill dey produce two things: KV cache (the key and value tensors) for every prompt token, and the first output token. Nothing reach reader until that pass finish. Na why prefill time and time to first token (TTFT) dey nearly be the same measurement.

Prefill cost dey grow as prompt length increase. The linear part na matrix work for each layer. The quadratic part na attention, where every token dey attend to every token wey come before am, and this part start to matter for long context. So if you double the prompt, TTFT go at least double.

You fit observe this within one minute. Send 200-token prompt to your server, then send 2,000-token prompt, while you ask for the same number of output tokens each time. TTFT go rise sharply. The streaming speed after the first token hardly change.

Why decode depend on memory bandwidth?

Decode dey produce one token for each step. To produce that one token, GPU must read every weight for the model from memory, use each weight for small number of operations, then throw the weight away. Arithmetic intensity dey close to 1, so compute units dey spend most of their time waiting.

Decode slow because each token need reading the whole model from memory. Na memory bus dey set the pace, while compute units dey idle.

This mean say the limit for single stream decode speed na the arithmetic wey you fit calculate on paper. Divide memory bandwidth by the number of bytes wey the weights occupy.

[
  {
    "device": "CPU, dual channel DDR5-5600",
    "mem_bandwidth_gb_s": 90,
    "decode_ceiling_tok_s": 6
  },
  {
    "device": "NVIDIA A10G",
    "mem_bandwidth_gb_s": 600,
    "decode_ceiling_tok_s": 38
  },
  {
    "device": "NVIDIA L40S",
    "mem_bandwidth_gb_s": 864,
    "decode_ceiling_tok_s": 54
  },
  {
    "device": "NVIDIA RTX 4090",
    "mem_bandwidth_gb_s": 1008,
    "decode_ceiling_tok_s": 63
  },
  {
    "device": "NVIDIA A100 80GB SXM",
    "mem_bandwidth_gb_s": 2039,
    "decode_ceiling_tok_s": 127
  },
  {
    "device": "NVIDIA H100 SXM",
    "mem_bandwidth_gb_s": 3350,
    "decode_ceiling_tok_s": 209
  }
]

The bandwidth column get the specification figure wey each vendor publish. The ceiling column na that figure divided by 16 GB, wey be the size of an 8 billion parameter model stored with 16 bit precision. Na arithmetic calculation be this, no be benchmark result. Your measured rate go dey below am, and knowing how far below e dey useful because e go tell you whether you need fix your serving stack or your hardware.

Read the 6 rows one after another, and the pattern clear. CPU for dual channel DDR5 dey move about 90 GB/s, wey limit decode to around 6 tokens per second for that model. L40S dey reach around 54. H100 SXM, with published bandwidth of 3350 GB/s, dey around 209.

Na this same reason quantization na the strongest single lever for decode speed. Store the same model with 8 bits instead of 16, and you go halve the bytes wey each token need read, so the ceiling go roughly double. You no add any compute. You just move less memory.

How I fit measure each phase for my own server?

Ollama dey return the split for the response body. Request non streaming completion, then read the counters.

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain memory bandwidth in two sentences.",
  "stream": false
}' | jq '{prompt_eval_count, prompt_eval_duration, eval_count, eval_duration}'

Use model tag wey you don actually pull, and ollama list go show you the tag. prompt_eval_count and prompt_eval_duration na prefill: prompt token count and time wey e spend on am. eval_count and eval_duration na decode. Durations dey nanoseconds, so decode speed na eval_count / eval_duration * 1e9 and prefill speed na prompt_eval_count / prompt_eval_duration * 1e9. Expect prefill rate to come out much higher than decode rate for the same request. Na this gap everything else here dey explain.

For OpenAI compatible server like vLLM, curl fit measure time to first byte for you.

curl -N -s -o /dev/null \
  -w 'pretransfer %{time_pretransfer}s  first_byte %{time_starttransfer}s\n' \
  http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Explain memory bandwidth.", "max_tokens": 128, "stream": true}'

time_starttransfer na the time wey the first body byte arrive, so with "stream": true e include TTFT plus connection setup. Subtract time_pretransfer to remove setup cost. Run am twice and keep the second result, because the first call fit include cold model load.

vLLM still publishes the split as Prometheus metrics for /metrics. Run curl -s http://localhost:8000/metrics | grep -E 'time_to_first_token|inter_token_latency' and you go get histograms vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds. Add vllm:num_requests_running and vllm:num_requests_waiting for queue depth, and vllm:kv_cache_usage_perc for cache pressure. Those five names na the complete dashboard.

Under load, vllm bench serve --model <name> --num-prompts 200 --request-rate 4 dey drive the running server and report time to first token plus latency for each output token with percentiles. Na only this way you fit see the two phases dey compete with each other. Before you tune anything, take clean baseline: the method for measuring tokens per second on local LLM go give you one wey survive reboot.

Why long system prompt dey delay first token but e no dey slow streaming speed?

Because system prompt na prefill work, and nothing more. System go process am once, for the same pass with the rest of the prompt, before first token appear. After that pass, e dey exist only as KV cache entries, and decode dey read dem together with everything else. So 3,000 token system prompt dey add to TTFT for every single request, while tokens per second nearly remain the same.

Nearly, but no be exactly. Decode dey read those extra KV entries again for every step, so very long prompt dey slow decode small. Next section go explain this.

The fix na to stop recomputing the same prefix. Server wey get prefix caching dey keep KV cache for shared prefix and reuse am, so second request wey carry the same system prompt go skip that part of prefill completely. vLLM dey call this automatic prefix caching; check vllm serve --help for your version, because default setting don change across releases. That in-GPU KV cache different from the prompt cache wey API provider dey bill you for, and the difference between KV cache and prompt cache worth reading before you tune either one.

Wetin make decode dey slow as context dey fill?

Two reasons dey involved, and both concern the KV cache.

The first one na bandwidth. For every decode step, attention dey read the keys and values of every token wey come before. Weights get fixed cost per token. The KV cache dey grow. You fit calculate the size from the model config.json: bytes per token equal 2 multiplied by num_hidden_layers, by num_key_value_heads, by the head dimension (hidden_size divided by num_attention_heads), by the bytes per element. The first 2 count one key and one value.

For common 8 billion parameter layout, with 32 layers, 8 key and value heads under GQA (grouped query attention), head dimension 128, and 16 bit precision, na 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB for each token. So, conversation wey get 8,000 tokens go carry roughly 1 GB KV cache for each request.

The second one na capacity. That 1 GB na memory wey no fit hold weights or another user's context. The server dey set the size of its KV pool once when e start, for vLLM through --gpu-memory-utilization, and when the pool full, new requests go wait. If vllm:num_requests_waiting dey climb while vllm:kv_cache_usage_perc dey near 1, na the exact sign of this condition. Some stacks dey preempt request wey dey run and recompute its cache later instead of queueing am. For the user, this dey look like stream dey pause halfway.

Long context cost you two times: e need more prefill work for the beginning, and e need more memory read for each token for the rest of the answer.

Wetin make batching improve throughput but make tail latency worse?

Because decode dey bandwidth bound, extra requests almost free for compute side. One read of the weights fit produce one token for every sequence inside the batch, so total throughput dey rise almost linearly as batch size increase, until either KV pool finish or batch become big enough to become compute bound again. Continuous batching dey rebuild batch every step, so finished request comot and queued request join without waiting for other requests wey dey inside the batch.

The cost dey show for percentiles. Each user next token now wait for the slowest part of shared step. So p50, wey be the median, still fit remain acceptable, while p99, wey be the slowest 1 request out of 100, dey stretch. Na p99 people dey notice, because na the pause wey happen for middle of sentence.

Prefill dey make the effect stronger. When big prompt enter while stream dey run, e occupy the device for one long step, and everybody wey dey stream at that time see a gap. Chunked prefill reduce most of this problem by cutting long prompt into pieces and mixing each piece inside decode batches. As of August 2026, vLLM V1 engine dey do this by default and expose the balance through --max-num-batched-tokens. vLLM tuning documentation explain the tradeoff clearly: smaller values, around 2048, give better inter token latency (ITL) because fewer prefills interrupt decodes, while larger values give better TTFT because more prefill tokens fit inside one batch. That single flag na prefill versus decode, exposed as number wey you fit adjust. The point where p99 stop being acceptable na capacity question, and how many concurrent users one self-hosted LLM fit serve dey work through am with the same metrics.

Why bigger GPU sometimes no dey change anything?

Because bigger usually mean more compute, and decode no need plenty compute.

Compare two rows for the chart above. A100 80GB get published bandwidth of 2039 GB/s, while L40S get 864 GB/s. The decode ceiling follow exactly: 127 tokens per second against 54. RTX 4090 na very fast card by most measurements, and e 1008 GB/s puts the ceiling at 63. No matter the other differences between two cards, single stream decode dey follow the bandwidth line for the specification sheet.

So, na two ways dey to make decode faster: read fewer bytes per token (quantize the weights, or run smaller model), or buy more bandwidth. Prefill na the opposite case. E need compute, so faster card really fit reduce TTFT for long prompts. If the complaint be say first token dey take four seconds, better hardware fit fix am. If the complaint be say the text dey type slowly, e probably no go fix am.

You suppose run prefill and decode for separate workers?

The big serving stacks dey do exactly this, and dem dey call the technique prefill and decode disaggregation. One pool of workers dey run prefill only, while second pool dey run decode only. The KV cache wey the first pool build dey transfer go the second pool through fast interconnect. E dey work because each phase need different hardware and different scheduling. Prefill need compute and big token batches. Decode need bandwidth and plenty concurrent sequences. When you split dem, each pool fit scale by itself, and one very big prompt no go stall every active stream.

For one VPS (virtual private server) with one GPU, e almost never worth am. You go dey divide one device against itself, and you go turn one pointer into network transfer of gigabytes of cache. The technique become useful when you get enough accelerators to dedicate complete machines to each phase, plus enough steady traffic to keep both pools busy. Below that level, chunked prefill fit provide almost the same isolation with one flag.

Wetin to change when the number no good

When TTFT too high:

• Make the prompt shorter. Prefill cost dey follow prompt tokens, and system prompt cost dey apply for every request.
• Turn on prefix caching so system go compute repeated prefix once instead of every time.
• Increase --max-num-batched-tokens so more prefill work go enter each step.
• Check the queue before you blame the model. vllm:num_requests_waiting wey pass zero mean say request never start, and na capacity problem be that.

When tokens per second too low:

• Quantize the weights. Fewer bytes per weight mean fewer bytes to read for each token.
• Check your card published memory bandwidth against the chart above, then see how close you dey to the ceiling.
• Reduce --max-num-batched-tokens so prefill no interrupt decode too often.
• Check the context length. When conversation don grow reach thousands of tokens, system dey read much bigger KV cache for every step.

The runtime matter here too, because Ollama and vLLM dey schedule prefill and decode differently, and setting wey help one fit do nothing for the other. Measure first for both phases, then change one thing.

FAQ

Why my first token dey take seconds but the rest dey stream fast?

The wait na prefill, and the streaming na decode. Prefill process the whole prompt for one compute bound pass before any output dey available, so the cost dey grow as prompt length increase. Decode then dey release one token per step at a rate wey memory bandwidth set, and this rate almost no depend on how long the prompt be. Long system prompt for every request na the usual cause. Prefix caching remove the repeated part of this cost.

Longer prompt dey slow down tokens per second?

Small, and the reason different from TTFT. Every decode step dey read the keys and values of all previous tokens, so bigger KV cache mean more bytes to read per token. For common 8 billion parameter layout, the cache na about 128 KiB per token, so 8,000 token context na roughly 1 GB wey system dey touch for every step. The bigger effect of long prompt still dey on TTFT, no be streaming speed.

Which GPU specification dey predict decode speed?

Memory bandwidth. Divide the published bandwidth by the size of the weights for memory, and you get the arithmetic ceiling for one stream. Card wey get more compute but the same bandwidth no go stream faster. Na this same reason quantizing to 8 bits dey roughly double decode speed: e cut the bytes wey system read per token by half without changing compute.

Why throughput dey increase when I add users but each user dey feel slower?

One read of the weights dey serve one token for every sequence inside the batch, so total tokens per second dey rise as batch size increase. Each user token now dey wait for one shared step, so latency per user dey rise at the same time. Monitor p99 inter token latency, no be the aggregate throughput number, and check vllm:num_requests_waiting to see whether requests dey queue instead of running.