Prefill vs decode: why token one lags

Prefill vs decode, in one paragraph

Prefill vs decode is the one distinction that explains most latency questions about a self-hosted LLM (large language model). Prefill reads the whole prompt in a single pass and is limited by compute. Decode writes the answer one token at a time and is limited by memory bandwidth. Time to first token is a prefill number. Tokens per second is a decode number.

Both phases run on the same GPU (graphics processing unit), with the same weights, inside the same process, so it is natural to treat them as one workload. They behave like two different programs sharing one device. Separate them and a long list of confusing results stops being confusing.

Why is prefill compute bound?

Prefill pushes the entire prompt through every layer once. A 2,000 token prompt gives every matrix multiply 2,000 rows of work, so the GPU does a lot of arithmetic for each byte of weight it loads. That ratio, arithmetic per byte moved, is called arithmetic intensity, and prefill has a high one. The device runs near its compute limit and the memory bus has slack.

Prefill produces two things: the KV cache (the key and value tensors) for every prompt token, and the first output token. Nothing reaches the reader until that pass finishes, which is why prefill time and time to first token (TTFT) are close to the same measurement.

Prefill cost grows with prompt length. The linear part is the per layer matrix work. The quadratic part is attention, where each token attends to every earlier token, and it starts to matter at long context. So doubling the prompt at least doubles TTFT.

You can watch this in one minute. Send a 200 token prompt to your server, then a 2,000 token prompt, asking for the same number of output tokens each time. TTFT climbs sharply. The streaming speed after the first token barely moves.

Why is decode memory bandwidth bound?

Decode produces one token per step. To produce that single token the GPU must read every weight in the model out of memory, use each weight for a couple of operations, and discard it. Arithmetic intensity is close to 1, so the compute units spend most of their time waiting.

Decode is slow because each token requires reading the whole model out of memory, so the memory bus sets the pace and the compute units idle.

That makes the ceiling on single stream decode speed arithmetic you can do on paper. Divide memory bandwidth by the bytes the weights occupy.

[
  {
    "device": "CPU, dual channel DDR5-5600",
    "mem_bandwidth_gb_s": 90,
    "decode_ceiling_tok_s": 6
  },
  {
    "device": "NVIDIA A10G",
    "mem_bandwidth_gb_s": 600,
    "decode_ceiling_tok_s": 38
  },
  {
    "device": "NVIDIA L40S",
    "mem_bandwidth_gb_s": 864,
    "decode_ceiling_tok_s": 54
  },
  {
    "device": "NVIDIA RTX 4090",
    "mem_bandwidth_gb_s": 1008,
    "decode_ceiling_tok_s": 63
  },
  {
    "device": "NVIDIA A100 80GB SXM",
    "mem_bandwidth_gb_s": 2039,
    "decode_ceiling_tok_s": 127
  },
  {
    "device": "NVIDIA H100 SXM",
    "mem_bandwidth_gb_s": 3350,
    "decode_ceiling_tok_s": 209
  }
]

The bandwidth column holds each vendor's published specification figure. The ceiling column is that figure divided by 16 GB, the size of an 8 billion parameter model stored at 16 bit precision. It is arithmetic, not a benchmark result. Your measured rate will land below it, and knowing how far below is useful, because it tells you whether to fix your serving stack or your hardware.

Read the 6 rows in order and the pattern is clear. A CPU on dual channel DDR5 moves about 90 GB/s, which caps decode near 6 tokens per second for that model. An L40S lands near 54. An H100 SXM, with published bandwidth of 3350 GB/s, sits near 209.

This is also why quantization is the strongest single lever on decode speed. Store the same model at 8 bits instead of 16 and you halve the bytes read per token, so the ceiling roughly doubles. You added no compute. You moved less memory.

How do I measure each phase on my own server?

Ollama returns the split in the response body. Ask for a non streaming completion and read the counters.

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain memory bandwidth in two sentences.",
  "stream": false
}' | jq '{prompt_eval_count, prompt_eval_duration, eval_count, eval_duration}'

Use a model tag you have actually pulled, which ollama list will show you. prompt_eval_count and prompt_eval_duration are prefill: the prompt token count, and the time spent on it. eval_count and eval_duration are decode. Durations are in nanoseconds, so decode speed is eval_count / eval_duration * 1e9 and prefill speed is prompt_eval_count / prompt_eval_duration * 1e9. Expect the prefill rate to come out far higher than the decode rate on the same request. That gap is what everything else here explains.

For an OpenAI compatible server such as vLLM, curl can time the first byte for you.

curl -N -s -o /dev/null \
  -w 'pretransfer %{time_pretransfer}s  first_byte %{time_starttransfer}s\n' \
  http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Explain memory bandwidth.", "max_tokens": 128, "stream": true}'

time_starttransfer is the moment the first body byte arrived, so with "stream": true it is TTFT plus connection setup. Subtract time_pretransfer to remove the setup cost. Run it twice and keep the second result, because the first call can include a cold model load.

vLLM also publishes the split as Prometheus metrics on /metrics. Run curl -s http://localhost:8000/metrics | grep -E 'time_to_first_token|inter_token_latency' and you get the histograms vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds. Add vllm:num_requests_running and vllm:num_requests_waiting for queue depth, and vllm:kv_cache_usage_perc for cache pressure. Those five names are the whole dashboard.

Under load, vllm bench serve --model <name> --num-prompts 200 --request-rate 4 drives the running server and reports time to first token and per output token latency with percentiles, which is the only way to see the two phases fight each other. Before you tune anything, take a clean baseline: the method in measuring tokens per second on a local LLM gives you one that survives a reboot.

Why does a long system prompt delay the first token but not the streaming speed?

Because the system prompt is prefill work and nothing else. It is processed once, in the same pass as the rest of the prompt, before the first token appears. After that pass it exists only as KV cache entries, and decode reads those along with everything else. So a 3,000 token system prompt adds to TTFT on every single request while leaving tokens per second nearly unchanged.

Nearly, not exactly. Those extra KV entries are read again at every decode step, so a very long prompt does slow decode a little. The next section covers that.

The fix is to stop recomputing the same prefix. A server with prefix caching keeps the KV cache of a shared prefix and reuses it, so the second request carrying the same system prompt skips that part of prefill entirely. vLLM calls this automatic prefix caching; check vllm serve --help on your version, because the default has moved across releases. That in-GPU KV cache is a different thing from the prompt cache an API provider bills you for, and the difference between a KV cache and a prompt cache is worth reading before you tune either one.

Why does decode slow down as the context fills?

Two reasons, both about the KV cache.

The first is bandwidth. At every decode step, attention reads the keys and values of every previous token. Weights are a fixed cost per token. The KV cache is a growing one. You can compute its size from the model's config.json: bytes per token equals 2 multiplied by num_hidden_layers, by num_key_value_heads, by the head dimension (hidden_size divided by num_attention_heads), by the bytes per element. The leading 2 counts one key and one value.

For a common 8 billion parameter layout, 32 layers, 8 key and value heads under GQA (grouped query attention), head dimension 128, at 16 bit precision, that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. An 8,000 token conversation therefore carries roughly 1 GB of KV cache, per request.

The second is capacity. That 1 GB is memory which cannot hold weights or another user's context. The server sizes its KV pool once at startup, on vLLM through --gpu-memory-utilization, and when the pool is full new requests wait. vllm:num_requests_waiting climbing while vllm:kv_cache_usage_perc sits near 1 is the exact signature of that state. Some stacks preempt a running request and recompute its cache later instead of queueing, which the user experiences as a stall in the middle of a stream.

Long context costs you twice: more prefill work at the start, and more memory read per token for the rest of the answer.

Why does batching help throughput and hurt tail latency?

Because decode is bandwidth bound, extra requests are close to free on the compute side. One read of the weights can produce a token for every sequence in the batch, so total throughput rises almost linearly with batch size until either the KV pool runs out or the batch grows large enough to become compute bound again. Continuous batching rebuilds the batch every step, so a finished request leaves and a queued one joins without waiting for its neighbours.

The bill arrives in the percentiles. Each user's next token now waits for the slowest part of a shared step, so p50, the median, stays acceptable while p99, the slowest 1 request in 100, stretches. p99 is what people notice, because it is the pause in the middle of a sentence.

Prefill makes it sharper. A large prompt arriving mid stream occupies the device for one long step, and everybody currently streaming sees a gap. Chunked prefill removes most of that by cutting a long prompt into pieces and mixing each piece into the decode batches. As of August 2026 the vLLM V1 engine does this by default and exposes the balance through --max-num-batched-tokens. The vLLM tuning documentation states the tradeoff plainly: smaller values, around 2048, give better inter token latency (ITL) because fewer prefills interrupt decodes, and larger values give better TTFT because more prefill tokens fit into one batch. That single flag is prefill versus decode, exposed as a number you can turn. Where the p99 stops being acceptable is a capacity question, and how many concurrent users one self-hosted LLM can serve works through it with the same metrics.

Why does a bigger GPU sometimes change nothing?

Because bigger usually means more compute, and decode does not want compute.

Compare two rows of the chart above. The A100 80GB carries published bandwidth of 2039 GB/s against the L40S at 864 GB/s, and the decode ceiling follows exactly: 127 tokens per second versus 54. The RTX 4090 is a very fast card by most measures and its 1008 GB/s puts its ceiling at 63. Whatever else differs between two cards, single stream decode tracks the bandwidth line on the specification sheet.

So there are two ways to make decode faster: read fewer bytes per token (quantize the weights, or run a smaller model), or buy more bandwidth. Prefill is the opposite case. It wants compute, so a faster card genuinely shortens TTFT on long prompts. If the complaint is that the first token takes four seconds, better hardware may fix it. If the complaint is that the text types slowly, it probably will not.

Should you run prefill and decode on separate workers?

The large serving stacks do exactly this, and the technique is called prefill and decode disaggregation. One pool of workers runs prefill only, a second pool runs decode only, and the KV cache built by the first is transferred to the second over a fast interconnect. It works because the phases want different hardware and different scheduling. Prefill wants compute and big token batches. Decode wants bandwidth and many concurrent sequences. Splitting them lets each pool scale on its own, and it stops one huge prompt from stalling every active stream.

On a single VPS (virtual private server) with one GPU it is almost never worth doing. You would be dividing one device against itself, and you would turn a pointer into a network transfer of gigabytes of cache. The technique pays off once you have enough accelerators to dedicate whole machines to each phase, and enough steady traffic to keep both pools busy. Below that, chunked prefill buys most of the same isolation for one flag.

What to change when the number is bad

When TTFT is too high:

• Shorten the prompt. Prefill cost tracks prompt tokens, and the system prompt is paid on every request.
• Turn on prefix caching so a repeated prefix is computed once instead of every time.
• Raise --max-num-batched-tokens so more prefill work lands in each step.
• Check the queue before blaming the model. vllm:num_requests_waiting above zero means the request had not started, which is a capacity problem.

When tokens per second is too low:

• Quantize the weights. Fewer bytes per weight is fewer bytes read per token.
• Check your card's published memory bandwidth against the chart above and see how close you are to the ceiling.
• Lower --max-num-batched-tokens so prefills interrupt decode less often.
• Check the context length. A conversation grown to thousands of tokens reads a much larger KV cache at every step.

The runtime matters here too, because Ollama and vLLM schedule prefill and decode differently, and a setting that helps one can do nothing on the other. Measure first, in both phases, then change one thing.

FAQ

Why does my first token take seconds but the rest streams fast?

The wait is prefill, and the streaming is decode. Prefill processes the whole prompt in one compute bound pass before any output exists, so its cost grows with prompt length. Decode then emits one token per step at a rate set by memory bandwidth, which is almost independent of how long the prompt was. A long system prompt on every request is the usual cause. Prefix caching removes the repeated part of that cost.

Does a longer prompt slow down tokens per second?

A little, and for a different reason than TTFT. Every decode step reads the keys and values of all previous tokens, so a bigger KV cache means more bytes read per token. For a common 8 billion parameter layout the cache is about 128 KiB per token, so an 8,000 token context is roughly 1 GB that gets touched at every step. The larger effect of a long prompt is still on TTFT, not on streaming speed.

Which GPU specification predicts decode speed?

Memory bandwidth. Divide the published bandwidth by the size of the weights in memory and you have the arithmetic ceiling for one stream. A card with more compute but the same bandwidth will not stream faster. That is also why quantizing to 8 bits roughly doubles decode speed: it halves the bytes read per token without touching compute.

Why does throughput go up when I add users but each user feels slower?

One read of the weights serves a token for every sequence in the batch, so total tokens per second rises with batch size. Each individual token now waits for a shared step, so per user latency rises at the same time. Watch p99 inter token latency, not the aggregate throughput number, and check vllm:num_requests_waiting to see whether requests are queueing rather than running.