Why Your Self-Hosted LLM Dey Slow With 5 Users

Why self-hosted LLM slow down when more users come?

Self-hosted LLM dey stall for 5 concurrent users because server still dey generate one reply at a time, while the other four dey queue. Ollama documentation talk am plainly about the default: OLLAMA_NUM_PARALLEL na “the maximum number of parallel requests each model will process at the same time, default 1.” Nothing spoil. Four out of your five users dey wait for their turn.

The fix rarely be bigger server. Na serving engine wey fit push many requests through the model for the same forward pass, plus enough spare memory to hold everybody conversation while e dey work. Both parts matter, and na the second one actually set your limit.

The two phases wey every request dey pass through

Prefill dey read the whole prompt at once and build attention cache for am. Every prompt token dey pass through the model together, so prefill na one big matrix multiply, and arithmetic throughput dey limit am. Decode then dey write the answer one token at a time. Every token need make the model read all im weights from memory again, while the arithmetic wey happen for that one token small well-well. Memory bandwidth dey limit decode.

Na this difference be the main reason batching dey work. Decode for one user fit read, for example, 5 GB of weights for every token and leave most arithmetic units idle. Add second request, and the engine go read the same 5 GB once, then calculate two tokens from am. The second user nearly no add extra time. If you serve requests strictly one after another, you lose this benefit.

Two numbers describe wetin user dey experience. TTFT (time to first token) na queue wait plus prefill. ITL (inter-token latency) na the gap between streamed tokens, and decode dey determine am. Slow server usually slow for one of these two areas, and the fixes no be the same.

Static batching dey make everybody wait for the slowest reply

Static batching na the naive version. Na wetin you get when you group requests by yourself for application code. The engine go collect N requests, run dem together, and hold every slot until the longest generation for the group finish.

One user wey ask for 1,200 token summary fit keep four one-line answers locked inside the batch, because the batch no go release any slot until the slowest member finish.

Two costs dey follow. Finished sequences still dey occupy slots wey no dey do any useful compute, so effective throughput dey drop as output lengths vary. Chat output lengths dey vary plenty. Request wey arrive one step after the batch form go wait for the whole batch to drain before e even start prefill. This mean say na another person essay go determine im TTFT.

Continuous batching dey admit and retire requests every token

Continuous batching dey schedule for level of one decoding step. After each step, scheduler go remove sequences wey just emit their stop token, then admit requests wey dey wait into free slots. Reply wey end for step 40 go free its slot for step 40, no be when batch finish.

This no be anything unusual. llama-server document -cb, --cont-batching as "whether to enable continuous batching (a.k.a dynamic batching) (default: enabled)", and vLLM na around this idea. Ollama sef fit serve parallel requests. Default setting just limit the number to one. Na why plenty people conclude say their hardware no fit handle concurrency, when na their configuration talk no.

People usually measure published continuous batching results on datacenter cards wey get spare compute and tens of gigabytes for the cache. The pattern for those results still apply to your box. But their size no apply, and the memory section below explain why.

Prefill dey compete with decode for the same compute

When new request enter while four replies dey stream, e first need prefill the prompt, and prefill dey use plenty compute. If scheduler give that prefill one step by itself, the four users wey dey stream no go receive any token during that time. For long prompt, every open window go show that pause clearly. Na this stutter people dey talk about when dem say server dey hiccup anytime another person press send.

Chunked prefill dey break long prompt into pieces, then mix each piece with the running decodes for the same step. vLLM tuning guide state the tradeoff directly: smaller chunk budgets "achieve better ITL because there are fewer prefills slowing down decodes", while higher values "achieve better time to first token (TTFT) as you can process more prefill tokens in a batch". You dey choose whose experience to protect: the person wey dey wait for reply to start, or the people wey dey watch text stream.

Prompt length decide how much this issue go affect you. A 6,000 token prompt with 200 token answer mean 6,000 tokens of prefill work against 200 decode steps. Retrieval-augmented chat and long system prompts both dey push you enter this situation, so prefill no longer be small rounding error; na the thing users dey wait for. Prefix caching dey help when the long part dey repeat: vLLM exposes --enable-prefix-caching, wey reuse cache for shared prompt prefix instead of recomputing am for every request.

Memory wey go finish first na KV cache

Every token for every active conversation dey leave one key vector and one value vector for every layer of the model. Na this be KV cache (key/value cache), and na e dey allow decode avoid recompute the whole prompt for every new token. The size per token dey fixed by the model shape: 2 (one key, one value) times the layer count, times the number of key/value heads, times the head dimension, times the bytes per value. Read those numbers from the model config.json.

Calculate am once, and the limit no go remain mystery. One normal 8B model wey get 36 layers, 8 key/value heads, and head dimension of 128, while e hold the cache for 16-bit, go use 2 36 8 128 2 bytes per token. That na 147,456 bytes, about 144 KiB. One 8,192-token conversation therefore need roughly 1.2 GB cache. Five of dem need roughly 6 GB, on top of the weights. Na this be the real answer to how many users fit run.

Concurrency dey multiply context, and the tools dey show am clearly. Ollama FAQ talk say: "Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation." RAM wey you need dey scale with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH. For llama-server, the context wey you request with -c dey share across the -np slots. So, if you increase the slot count by itself, each request go get smaller space to hold content. Read the context per slot from the startup log instead of assuming am.

vLLM dey preallocate instead. --gpu-memory-utilization (default 0.92) na "the fraction of GPU memory to be used for the model executor". After the weights, anything wey remain become the paged KV pool. When that pool no get enough space, the scheduler go evict one request instead of failing am:

WARNING 05-09 00:49:33 scheduler.py:1057] Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

For vLLM V1 engine, the default preemption mode na RECOMPUTE. So, when e evict one request, e discard the cache and prefill am again when e admit the request back. That work dey happen two times. The documentation warn say "preemption and recomputation can adversely affect end-to-end latency", and this log line na the clearest explanation for why one unlucky user wait much longer than everybody else while your average latency still look healthy. Set disable_log_stats=False to log the cumulative count, or read the preemption counter from the Prometheus metrics wey vLLM dey expose.

Wetin change for 2, 5 and 20 concurrent users

Two users. For GPU wey still get cache space, you almost no go notice am, because the second decode stream dey run alongside the first one with very small extra time. But for CPU-only VPS wey get 4 to 8 GB of RAM, e no be free: both streams dey share the same small number of vCPUs and the same RAM bandwidth. So each user go see roughly half the tokens per second, while cache demand go double against much smaller available space.

Five users. Na here default settings stop to dey enough, and the first problem na queue. With OLLAMA_NUM_PARALLEL for 1, four people go wait for whoever request the long answer, and each person go see normal speed as soon as e reach their turn. If you increase the parallel count, the problem go change: five slots with 8K context each need 40K-token cache space. If e no fit inside VRAM, the engine go offload layers to system RAM. If e no fit inside RAM, the machine go use swap and tokens per second go drop badly.

Twenty users. Twenty people for chat UI usually no mean twenty concurrent requests. Na this point dey important to understand before you buy hardware. Person fit read reply and think for 20 to 60 seconds before the next turn, so most of the session dey idle. But twenty agents, or twenty document summarisation jobs, na twenty real streams wey no get idle time at all. Na different kind of machine dem need.

Your users dey make requests at the same time, or dem just dey logged in?

Work out requests wey dey in flight before you size anything. The arithmetic simple: requests in flight equals users, times seconds wey e dey take to generate each turn, divided by seconds between turns.

1. First measure your own speed for one stream, both prefill and decode. No borrow number from another person card: measure tokens per second for your own box and use the result.
2. Estimate the duty cycle. Twenty chat users, 12 seconds of generation for each turn, and one turn every 90 seconds gives 20 * 12 / 90, wey be about 2.7 requests in flight.
3. Set the slot count small above that number, then check am against memory: slots times per-request context must fit inside the cache tokens wey you actually get.
4. Keep the queue short so overflow go fail fast and show clearly.

Available cache tokens na free memory after the weights, divided by the per-token cost from the section above. A 24 GB card wey dey run an 8B model for 16-bit uses about 16 GB for weights and gets roughly 6 GB usable cache with the default utilisation. That one na around five 8K conversations. To fit more, reduce the per-request context, or store the cache for 8-bit (llama-server takes --cache-type-k q8_0). Both options increase concurrency by giving up something, and you suppose read the honest explanation of that trade before you spend money on hardware: where GPU VPS breaks even against API tokens.

Wey Ollama default settings no longer dey enough

Increase the parallel count through the service unit, because shell export no go reach daemon wey systemd dey manage.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_MAX_QUEUE=64"

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama --property=Environment
ollama ps

systemctl show suppose print the three variables wey you just set. If e no do am, the drop-in no save, and nothing else wey you do go matter. ollama ps then list the loaded model with size wey pass the weights alone, because four slots of 8,192 tokens reserve 32,768 tokens of cache beside dem. If PROCESSOR column show say part of the model dey CPU when you expect everything dey GPU, e mean say you request more cache than the card get available. Reduce one of the two numbers.

The queue default need another check. Ollama dey queue up to OLLAMA_MAX_QUEUE requests, and "the default is 512". After that, e responds "with a 503 error indicating the server is overloaded". Queue wey deep reach 512 for machine wey dey serve four requests at once na promise wey you no fit keep, because client for position 300 go timeout long before na im turn. Short queue go return error wey your application fit retry or report, and that better pass spinner wey no ever resolve.

Test am for real. Send two requests at the same time from two terminals and monitor both. If the second one produce nothing until the first one finish, the parallel setting no take effect.

When real serving engine dey pay for itself

vLLM dey worth the extra setup when you get GPU wey still get spare capacity and more than roughly four requests dey genuinely in flight. Its scheduler dey work per token, its cache dey paged so e fit reuse free fragments, and e dey turn spare VRAM to concurrency instead of leaving am idle. As of August 2026, the documented install and launch na two commands:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [{"role": "user", "content": "Say hello."}]
    }'

Reply wey contain a choices array mean say server dey up and model don load. Under load, the two knobs wey matter na --max-num-seqs, the "maximum number of sequences to be processed in a single iteration", and --max-num-batched-tokens, the "maximum number of tokens that can be processed in a single iteration". The first one dey limit concurrency. The second one na the chunked prefill budget wey we describe earlier.

Below about four requests in flight, or for any box wey no get supported GPU, vLLM dey add complexity but e no dey give much benefit. E expect CUDA-class card and e dey claim most of the memory when e start, so this trade no make sense for 4 to 8 GB VPS. For there, use smaller model with shorter context and queue wey you control. how Ollama and vLLM differ as serving engines explain the choice fully, and running Qwen 3 8B on a VPS show wetin mid-sized model need before you add even one extra user.

The tradeoff wey folklore dey hide

Continuous batching dey increase total throughput, and e usually improve median latency too, because queued request fit start earlier. Tail latency dey move for the other direction, and na this part dem hardly mention.

Every extra sequence wey enter one step dey add small work, so ITL dey rise for everybody as batch dey fill up. When new request arrive, e prefill go take part of one step wey streaming users for don get otherwise. When cache pressure happen, scheduler fit preempt, and this go send request wey don generate part of output back to the beginning of prefill.

Chat UI dey show tail latency, no be average. Stream wey pause for two seconds in the middle of sentence go look broken, even when total time to completion good. Measure p95 TTFT and p95 ITL under the load wey you expect, and treat mean tokens per second as capacity number, no be description of the user experience.

The practical setting follow from this. Cap concurrency small below wetin memory fit handle, so engine no ever need preempt. Short queue wey predictable better pass deep batch wey dey thrash, because user wey wait four seconds then stream smoothly go happier pass user wey start immediately but stream pause two times.

Wetin to check when e slow

Every user normal, but dem dey wait long. Na queue be that, no be speed problem. Check the parallel setting first. The model dey serve correctly, one request at a time.

HTTP 503 from Ollama. Queue don full. Either the box really don reach capacity, or OLLAMA_MAX_QUEUE dey set low on purpose to shed load, and na exactly wetin you want am to do.

Tokens per second dey drop badly under load for CPU box. Run vmstat 1 while e dey happen. If si and so columns get nonzero values, the machine dey swap. That means e dey read weights from disk for every token. No configuration change fit save that. Reduce the model size or the slot count.

One user out of ten dey wait much longer than the others. Search the vLLM log for preempted. Preemption and the recompute wey follow am na the usual cause. E means cache don get more demand than the context length wey you allow fit handle.

TTFT bad even when server idle. Na prefill be the problem, no be concurrency. Long prompts dey take real time before the first token show, so check prompt size and prefix caching before you check hardware.

FAQ

Why self-hosted LLM dey slow down when second person use am?

Most time e no dey slow down at all. E dey queue. Ollama ships with OLLAMA_NUM_PARALLEL set to 1, so the second request dey wait for the first one to emit its final token. To tell the two cases apart, time one user's stream while another user dey wait: if their tokens per second normal once dem start, na queue you get, and raising the parallel count go fix am. If both streams dey run at half speed, una dey genuinely share memory bandwidth, and na hardware limit be that.

How many concurrent users fit one small GPU serve?

Count memory, no be users. Weights come first, then KV cache, wey cost 2 times layers times key/value heads times head dimension times bytes, per token, for each active conversation. A typical 8B model with 36 layers, 8 key/value heads and head dimension 128 cost about 144 KiB per token for 16-bit, so 8,192 token conversation need roughly 1.2 GB. A 24 GB card wey dey hold that model for 16-bit get about 6 GB left for cache. That one fit hold about five conversations with full context, or more if you make the context shorter.

Continuous batching dey make each user's reply slower?

Median latency usually dey improve, because requests no longer wait for the whole batch to finish. Tail latency dey worse. Each extra sequence add work to every decoding step. A new arrival's prefill dey take part of one step from users wey dey stream. A request wey get preempted must do prefill twice. Measure p95 inter-token latency, no be the average, because chat window dey make pauses obvious in a way wey averages dey hide.

Make I raise OLLAMA_NUM_PARALLEL or move to vLLM?

Raise the parallel count first. E free and e only need one drop-in file. E fix the common case where four people dey queue behind one long answer. Memory na the limit: parallel requests multiply the context wey you must hold, so monitor for layers wey dey spill go CPU. Move to vLLM when you get GPU with spare VRAM and more than about four requests truly dey in flight. Na for that point paged cache and per-token scheduling dey return more benefit than wetin dem cost.

More CPU cores go fix slow LLM server?

No be for the part wey users notice pass. Decode dey read the whole model from memory for each token, so RAM bandwidth dey limit am. Extra cores stop to help once bandwidth don saturate. Prefill dey scale with cores, so more cores dey reduce time to first token for long prompts. For 4 to 8 GB VPS, memory capacity usually na the binding constraint. The effective fix na smaller model or shorter context, no be more vCPUs.