Why your self-hosted LLM stalls at 5 users

Why does a self-hosted LLM slow down when more users arrive?

A self-hosted LLM stalls at 5 concurrent users because the server is still generating one reply at a time, and the other four are standing in a queue. Ollama's documentation is blunt about the default: OLLAMA_NUM_PARALLEL is "the maximum number of parallel requests each model will process at the same time, default 1." Nothing is broken. Four of your five people are waiting for a turn.

The repair is rarely a bigger box. It is a serving engine that pushes many requests through the model in the same forward pass, plus enough spare memory to hold everyone's conversation while it does. Both halves matter, and the second one is what actually sets your ceiling.

The two phases every request goes through

Prefill reads the whole prompt at once and builds the attention cache for it. Every prompt token goes through the model together, so prefill is one large matrix multiply, and it is limited by arithmetic throughput. Decode then writes the answer one token at a time. Each token needs the model's full weights read out of memory again, while the arithmetic done on that single token is tiny. Decode is limited by memory bandwidth.

That asymmetry is the whole reason batching works. Decoding for one user reads, say, 5 GB of weights per token and leaves most of the arithmetic units idle. Add a second request and the engine reads the same 5 GB once, then computes two tokens from it. The second user costs almost no extra time. Serving requests strictly one after another throws that away.

Two numbers describe what a user feels. TTFT (time to first token) is queue wait plus prefill. ITL (inter-token latency) is the gap between streamed tokens, and it is set by decode. A slow server is usually slow in one of these, and the fixes are not the same.

Static batching makes everyone wait for the slowest reply

Static batching is the naive version, and it is what you get if you group requests yourself in application code. The engine collects N requests, runs them together, and holds every slot until the longest generation in the group finishes.

One user asking for a 1,200 token summary keeps four one-line answers locked in the batch, because the batch releases no slot until its slowest member is done.

Two costs follow. Finished sequences keep occupying slots that compute nothing useful, so effective throughput drops as output lengths vary, and chat output lengths vary a lot. A request that arrives one step after the batch formed waits for the entire batch to drain before it even starts prefill, which means its TTFT is set by somebody else's essay.

Continuous batching admits and retires requests every token

Continuous batching schedules at the level of a single decoding step. After each step the scheduler drops sequences that just emitted their stop token, then admits waiting requests into the free slots. A reply that ends at step 40 frees its slot at step 40, not at the end of a batch.

This is not exotic. llama-server documents -cb, --cont-batching as "whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)", and vLLM is built around the idea. Ollama serves parallel requests too. The default just caps the number at one, which is why so many people conclude their hardware cannot do concurrency when it was their configuration that said no.

Published continuous batching results are usually measured on datacenter cards that have both spare compute and tens of gigabytes for the cache. The shape of those results carries over to your box. The size of them does not, and the memory section below is why.

Prefill competes with decode for the same compute

When a new request lands while four replies are streaming, its prompt has to be prefilled first, and prefill is compute heavy. If the scheduler gives that prefill a step of its own, the four streaming users receive no token during it. On a long prompt that is a visible pause in every open window. This is the stutter people mean when they say the server hiccups whenever somebody else hits send.

Chunked prefill breaks a long prompt into pieces and mixes each piece into the same step as the running decodes. vLLM's tuning guide states the tradeoff directly: smaller chunk budgets "achieve better ITL because there are fewer prefills slowing down decodes", while higher values "achieve better time to first token (TTFT) as you can process more prefill tokens in a batch". You are choosing whose experience to protect: the person waiting for a reply to begin, or the people watching text stream.

Prompt length decides how much this hurts. A 6,000 token prompt with a 200 token answer is 6,000 tokens of prefill work against 200 decode steps. Retrieval-augmented chat and long system prompts both push you into that regime, so prefill stops being a rounding error and becomes the thing users wait on. Prefix caching helps when the long part repeats: vLLM exposes --enable-prefix-caching, which reuses the cache for a shared prompt prefix instead of recomputing it for every request.

The memory that runs out first is the KV cache

Every token in every active conversation leaves a key vector and a value vector in every layer of the model. That is the KV cache (key/value cache), and it is what lets decode avoid recomputing the whole prompt for each new token. Its size per token is fixed by the model's shape: 2 (one key, one value) times the layer count, times the number of key/value heads, times the head dimension, times the bytes per value. Read those numbers out of the model's config.json.

Work it once and the ceiling stops being a mystery. A typical 8B model with 36 layers, 8 key/value heads and a head dimension of 128, holding the cache in 16-bit, costs 2 36 8 128 2 bytes per token. That is 147,456 bytes, about 144 KiB. One 8,192 token conversation therefore needs roughly 1.2 GB of cache. Five of them need roughly 6 GB, on top of the weights, and that is the real answer to how many users fit.

Concurrency multiplies context, and the tools say so out loud. Ollama's FAQ: "Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation." Required RAM scales with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH. In llama-server, the context you ask for with -c is shared out across the -np slots, so raising the slot count on its own shrinks what each request can hold. Read the per-slot context from the startup log instead of assuming it.

vLLM preallocates instead. --gpu-memory-utilization (default 0.92) is "the fraction of GPU memory to be used for the model executor". Whatever is left after the weights becomes the paged KV pool, and when that pool runs short the scheduler evicts a request rather than failing it:

WARNING 05-09 00:49:33 scheduler.py:1057] Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

In vLLM's V1 engine the default preemption mode is RECOMPUTE, so an evicted request discards its cache and prefills again when it is readmitted. That work is done twice. The documentation warns that "preemption and recomputation can adversely affect end-to-end latency", and this log line is the single best explanation for why one unlucky user waited far longer than everybody else while your average looked healthy. Set disable_log_stats=False to log the cumulative count, or read the preemption counter from the Prometheus metrics vLLM exposes.

What changes at 2, 5 and 20 concurrent users

Two users. Almost invisible on a GPU with cache to spare, because the second decode stream rides along with the first for very little extra time. On a CPU-only VPS with 4 to 8 GB of RAM it is not free: both streams share the same handful of vCPUs and the same RAM bandwidth, so each user sees roughly half the tokens per second, and cache demand doubles against a much smaller budget.

Five users. This is where defaults stop being enough, and it starts as a queue problem. With OLLAMA_NUM_PARALLEL at 1, four people wait on whoever asked for the long answer, and each of them sees normal speed the moment their turn arrives. Raise the parallel count and the problem changes shape: five slots at 8K context each is a 40K token cache to find. If it does not fit in VRAM the engine offloads layers to system RAM, and if it does not fit in RAM the box swaps and tokens per second collapses.

Twenty users. Twenty humans in a chat UI are usually not twenty concurrent requests, and this is the most useful thing to understand before buying hardware. A person reads a reply and thinks for 20 to 60 seconds between turns, so most of their session is idle. Twenty agents, or twenty document summarisation jobs, are twenty real streams with no idle time at all. That is a different machine.

Are your users concurrent, or only logged in?

Work out requests in flight before you size anything. The arithmetic is ordinary: in flight equals users, times seconds spent generating per turn, divided by seconds between turns.

1. Measure your own single-stream speed first, prefill and decode both. Do not borrow a number from someone else's card: measure tokens per second on your own box and use what you get.
2. Estimate the duty cycle. Twenty chat users, 12 seconds of generation per turn, one turn every 90 seconds, gives 20 * 12 / 90, which is about 2.7 requests in flight.
3. Set the slot count a little above that, then check it against memory: slots times per-request context must fit inside the cache tokens you actually have.
4. Keep the queue short so overflow fails fast and visibly.

Cache tokens available is free memory after the weights, divided by the per-token cost from the section above. A 24 GB card running an 8B model in 16-bit spends about 16 GB on weights and has roughly 6 GB of usable cache at the default utilisation, which is around five 8K conversations. To fit more, shorten the per-request context, or store the cache in 8-bit (llama-server takes --cache-type-k q8_0). Both buy concurrency by giving something up, and the honest version of that trade is worth reading before you commit money to hardware: where a GPU VPS breaks even against API tokens.

Where Ollama's defaults stop being enough

Raise the parallel count through the service unit, because a shell export will not reach a systemd-managed daemon.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_MAX_QUEUE=64"

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama --property=Environment
ollama ps

systemctl show should print the three variables you just set. If it does not, the drop-in did not save, and nothing else you do will matter. ollama ps then lists the loaded model with a size larger than the weights alone, because four slots at 8,192 tokens reserve 32,768 tokens of cache beside them. A PROCESSOR column showing part of the model on CPU when you expected all of it on GPU means you asked for more cache than the card had left. Lower one of the two numbers.

The queue default deserves a second look. Ollama queues up to OLLAMA_MAX_QUEUE requests, and "the default is 512". Past that it responds "with a 503 error indicating the server is overloaded". A 512-deep queue on a box that serves four at a time is a promise you cannot keep, because the client at position 300 times out long before its turn. A short queue returns an error your application can retry or report, which beats a spinner that never resolves.

Test it for real. Send two requests at the same moment from two terminals and watch both. If the second produces nothing until the first finishes, the parallel setting never took effect.

When a real serving engine starts paying for itself

vLLM earns its extra setup when you have a GPU with headroom and more than roughly four requests genuinely in flight. Its scheduler works per token, its cache is paged so free fragments get reused, and it converts spare VRAM into concurrency instead of leaving it idle. As of August 2026 the documented install and launch are two commands:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [{"role": "user", "content": "Say hello."}]
    }'

A reply containing a choices array means the server is up and the model is loaded. Under load the two knobs that matter are --max-num-seqs, the "maximum number of sequences to be processed in a single iteration", and --max-num-batched-tokens, the "maximum number of tokens that can be processed in a single iteration". The first caps concurrency. The second is the chunked prefill budget described earlier.

Below about four requests in flight, or on any box without a supported GPU, vLLM costs complexity and returns little. It expects a CUDA-class card and claims most of the memory at startup, which is the wrong trade on a 4 to 8 GB VPS. There the answer is a smaller model with a shorter context and a queue you control. how Ollama and vLLM differ as serving engines covers the choice in full, and running Qwen 3 8B on a VPS shows what a mid-sized model demands before you add a single extra user.

The tradeoff the folklore hides

Continuous batching raises total throughput, and it usually improves median latency too, because a queued request starts sooner. Tail latency moves the other way, and that half rarely gets mentioned.

Every extra sequence in a step adds a little work, so ITL rises for everyone as the batch fills. A new arrival's prefill takes a slice of a step the streaming users would otherwise have had. Under cache pressure the scheduler preempts, which sends a half-generated request back to the start of its prefill.

A chat UI shows tails, not averages. A stream that pauses for two seconds mid-sentence reads as broken even when total time to completion is good. Measure p95 TTFT and p95 ITL under the load you expect, and treat mean tokens per second as a capacity number rather than a description of the experience.

The practical setting follows from that. Cap concurrency slightly below what memory allows, so the engine never needs to preempt. A short predictable queue beats a deep batch that thrashes, because a user who waits four seconds and then streams smoothly is happier than a user who starts instantly and stalls twice.

What to check when it is slow

Each user is normal, the waiting is long. That is a queue, not a speed problem. Check the parallel setting first. The model is serving correctly, one request at a time.

HTTP 503 from Ollama. The queue is full. Either the box is genuinely at capacity, or OLLAMA_MAX_QUEUE is set low on purpose to shed load, which is what you want it to do.

Tokens per second collapses under load on a CPU box. Run vmstat 1 while it happens. Nonzero si and so columns mean the machine is swapping, so weights are being read from disk on every token. No configuration change saves that. Cut the model size or the slot count.

One user in ten waits far longer than the rest. Search the vLLM log for preempted. Preemption and its recompute is the usual cause, and it means the cache is oversubscribed for the context length you allow.

TTFT is bad even when the server is idle. That is prefill, not concurrency. Long prompts cost real time before the first token appears, so look at prompt size and prefix caching before you look at hardware.

FAQ

Why does my self-hosted LLM slow down when a second person uses it?

Most often it does not slow down at all. It queues. Ollama ships with OLLAMA_NUM_PARALLEL at 1, so the second request waits for the first to emit its final token. Tell the two cases apart by timing one user's stream while another waits: if their tokens per second is normal once they start, you have a queue, and raising the parallel count fixes it. If both streams run at half speed, you are genuinely sharing memory bandwidth, and that is a hardware limit.

How many concurrent users can one small GPU serve?

Count memory, not users. Weights first, then the KV cache, which costs 2 times layers times key/value heads times head dimension times bytes, per token, per active conversation. A typical 8B model with 36 layers, 8 key/value heads and head dimension 128 costs about 144 KiB per token in 16-bit, so an 8,192 token conversation needs roughly 1.2 GB. A 24 GB card holding that model in 16-bit has about 6 GB left for cache, which is about five conversations at full context, or more if you shorten the context.

Does continuous batching make each user's reply slower?

Median latency usually improves, because requests stop waiting for a whole batch to finish. Tail latency gets worse. Each additional sequence adds work to every decoding step, a new arrival's prefill steals part of a step from the streaming users, and a preempted request has to prefill twice. Measure p95 inter-token latency, not the average, because a chat window makes pauses obvious in a way that averages hide.

Should I raise OLLAMA_NUM_PARALLEL or move to vLLM?

Raise the parallel count first. It is free and takes one drop-in file, and it fixes the common case where four people queue behind one long answer. Memory is the limit: parallel requests multiply the context you must hold, so watch for layers spilling to CPU. Move to vLLM when you have a GPU with VRAM to spare and more than about four requests truly in flight, since that is the point where paged cache and per-token scheduling return more than they cost.

Will more CPU cores fix a slow LLM server?

Not for the part users notice most. Decode reads the whole model from memory for each token, so it is bound by RAM bandwidth, and extra cores stop helping once bandwidth is saturated. Prefill does scale with cores, so more of them shorten time to first token on long prompts. On a 4 to 8 GB VPS the binding constraint is usually memory capacity, and the effective fix is a smaller model or a shorter context rather than more vCPUs.