Ollama NUM_PARALLEL vs MAX_QUEUE: Wetin E Mean
See why your second Ollama request dey wait instead of fail, when HTTP 503 fit show, and how every OLLAMA_NUM_PARALLEL slot uses extra KV cache memory.
Wetin dey happen to the second Ollama request while the first one dey generate
Ollama concurrency dey depend on three environment variables, and by default, one loaded model dey serve one request at a time. The second request no dey get rejected, and e no dey receive partial answer. E go wait for queue until slot free, then e go run at normal speed.
Incoming request fit get three possible outcomes. E fit start immediately for free slot. E fit wait for queue. Or queue fit don full, and server go reject am with HTTP 503. OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS na dem dey decide which one go happen. But current releases get one important detail as of October 2026. FAQ dey describe queue in general terms, but scheduler source code dey more specific. HTTP 503 only apply to requests wey arrive while dem still need load their model, or reload am with different settings. Ollama no dey reject request wey an already loaded model fit serve as e be. That request go wait for slot for as long as e take.
Default setting safe, but na also why second user fit report say server "hang" when nothing spoil. To add slots, na two-line change. The main problem na memory. Every parallel slot need its own key/value cache (KV cache), wey be the memory block wey model dey keep for tokens wey e don process already. If you add slots without adding VRAM (video memory on the GPU), you fit turn slow answer into failed load. For CPU-only server, the same cache go use system RAM instead.
Wetin OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS each dey control
Na these be the default values for current Ollama releases as of October 2026. Check your own values instead make you trust the number wey dey here, by using the log line wey show later.
OLLAMA_NUM_PARALLELna how many requests one loaded model fit handle at the same time. The default na 1, so e dey serve requests one after another.OLLAMA_MAX_LOADED_MODELSna how many different models fit remain resident at the same time. The default na 0, meaning say Ollama go choose: three models per GPU, and three for machine wey no get GPU.OLLAMA_MAX_QUEUEna, as FAQ talk am, the maximum number of requests wey Ollama go queue when e busy before e reject more requests. The default na 512. If request enter when that queue don full, Ollama go reject am immediately. But for the current scheduler code, na only requests wey their model need load or reload first dey enter this queue. Requests for model wey don already load skip this queue and wait for free slot instead, and that wait no get limit. One maintainer don describe this as side effect of how dem implement the scheduler, and Ollama maintainers dey track fix for am as feature request (issue #7758), wey still dey open for October 2026.
The worst-case memory na the product of the first two. Two loaded models wey get four slots each mean eight KV cache slot allocations wey all remain resident at once, and Ollama go try satisfy am. For one GPU machine, e usually better make you hold one model and give am slots, because you fit do the arithmetic for your head.
Why every parallel slot costs VRAM
When Ollama loads a model, e dey launch one separate runner process. Two arguments wey e pass matter here: -c na the total context wey runner allocate KV cache for, while -np na the number of parallel sequences. Ollama set -c to your per-request context length multiplied by the slot count. Runner come divide that total equally among the slots, so every request still get the context length wey you ask for.
Na this be the full constraint, and na why parallelism no be free. If you move from one slot to four, you dey ask for four times the KV cache with the same per-request context. Nothing dey shared between slots, and one idle slot no fit lend its share to busy slot, because runner fix the split when e start. For GPU, that memory na VRAM. For CPU-only server, na system RAM, and the same multiplication still apply.
You fit read the real numbers instead of the ones wey you intend set:
journalctl -u ollama --no-pager -n 500 | grep "starting llama-server"That line hold the complete runner command line, including -c and -np. If -np be 1 after you set the variable, two possible causes dey. The first one na the model. Ollama limit some model architectures to one slot, no matter wetin you set, and e log model architecture does not currently support parallel requests when e do am. As of October 2026, that list include the Qwen 3.5, Qwen3-VL, Qwen3-Next, LFM2 and Nemotron-H families, and embedding models always get one slot. Run journalctl -u ollama --no-pager | grep "does not currently support parallel" to check. If that warning no dey, the setting no reach the server, and the next section explain why.
If the model weights plus that KV cache no fit inside VRAM, Ollama move some layers go system RAM, and those layers run on CPU. CPU layers slow pass GPU layers by far, so this make every request slower, including the single request wey you start with. Raising parallelism fit therefore reduce throughput instead of increasing am. With model wey big enough, the weights alone settle the matter before any slot calculation start. Na why self-hosting something wey big reach Kimi K3 na discussion about how many cards you get, instead of how many slots you set.
ollama psThe PROCESSOR column show 100% GPU when everything fit. A split like 35%/65% CPU/GPU mean say part of the model dey run on CPU. The SIZE column include the KV cache, so e grow when you raise the slot count and reload the model. Raise OLLAMA_NUM_PARALLEL, restart, send one request, and run ollama ps again: na the memory cost of your change be that, measured instead of guessed. If that measurement show say the model no longer fit, remember say the weights na the other half of the same budget, and moving from fp16 to a q8 or q4 build often free more VRAM than the slot wey you dey try add costs.
Context length and slot count multiply each other, so you need choose dem together. Large context with four slots na four large contexts. If you dey also tune the num_ctx context window for your model, change only one of the two at a time, or you no go know which one fill the card.
How to set these variables so dem survive reboot
For Linux, Ollama dey run as systemd service. If you run export OLLAMA_NUM_PARALLEL=4 for your shell, e no change anything, because systemd dey start the service with its own environment and e no dey see your shell. Use drop-in file.
sudo systemctl edit ollama.serviceAdd this inside the editor wey go open:
[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_MAX_QUEUE=32"Then reload and restart:
sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama --property=Environmentsystemctl show dey print wetin systemd go give the process. If your variable no dey there, drop-in file no save or daemon-reload no run. Confirm am from the server side too:
journalctl -u ollama --no-pager | grep "server config" | tail -1Ollama dey log the full environment when e start, for line wey message be server config. That map show exactly wetin the server read, and na the fastest way to settle whether variable take effect. E show the value wey you set, no be the number of slots wey particular model actually get. For that one, read the -np value for the runner line above.
Model wey don already load go keep the slot count wey e start with, because dem fix the value inside runner process when e launch. The restart above dey unload everything, so the next request go load the model again with the new setting and pay the load time once. How long model go remain loaded after that na separate control, and how to keep an Ollama model loaded between requests cover am.
Wetin served, queued and refused dey look like from client side
Send plenty requests at once and time dem. This one dey run eight streaming requests in parallel and print status plus timing for each one:
for i in $(seq 1 8); do
curl -s -o /dev/null \
-w "req$i http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
http://127.0.0.1:11434/api/generate \
-d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":true}' &
done
waitttfb na the time before the stream send the first byte. E dey close to time to first token (TTFT), because the first streamed chunk carry the first token.
Served in parallel. Every request dey report almost the same ttfb, and total dey rise for all of dem together. GPU dey share between the slots wey dey run, so each answer slow pass when e run alone, but more answers go finish every minute. Na this mode you dey buy when you increase OLLAMA_NUM_PARALLEL.
Queued. The first requests answer quickly, while the later ones show large ttfb before normal generation start. Na queue cause the wait, no be model. User wey dey watch chat window go see long blank pause, then text go dey come at full speed. This pattern — slow to start, then fast — show queue more than overloaded GPU.
Refused. Client go receive http=503 almost immediately, and the body na:
{"error":"server busy, please try again. maximum pending requests exceeded"}This message mean say scheduling queue full when the request arrive. E no talk anything about VRAM and e no talk anything about the model. For current releases, that queue only hold requests wey their model never load, or wey need reload with different settings. If model don already dey resident when you run the loop above, expect every request to return 200, no matter how low you set OLLAMA_MAX_QUEUE. The overflow go show as longer ttfb instead.
One honest limit: Ollama no publish queue depth. ollama ps and the /api/ps endpoint report the models wey don load, not the requests wey dey wait. So you go measure the queue from client side, by watching time to first byte, or count 503 responses for anything wey dey in front.
Why smaller MAX_QUEUE dey often better setting
Queue of 512 sound generous, and for one slot e nearly useless. Request 300 go wait behind 299 complete generations. That fit take minutes at best. Every HTTP client go give up long before then, so caller go see client-side timeout. This one no tell dem anything about the cause, and e no give your monitoring anything to alert on.
Smaller queue go turn that overflow into immediate 503, but na only for requests wey dey wait for model to load or reload, like burst wey come after restart or while models dey swap. As of October 2026, requests to model wey don already load go bypass OLLAMA_MAX_QUEUE and wait for free slot without limit. So, lowering the queue no go reduce wait time for busy server wey get one resident model. If you want fast refusal for that case, limit concurrency in front of Ollama, as the next section explain. 503 dey useful: reverse proxy fit retry am, client fit back off, dashboard fit count am, and person fit read am. Use your own measurements determine the cap. If one generation dey take about ten seconds and your client wait sixty, then about six requests per slot fit finish within that window. If you allow more than that to wait, na only timeouts you go produce.
When to put a queue in front of Ollama
Ollama queue wey dey inside na first in, first out (FIFO), and e no know who dey make the call. If na one application dey talk to one server, this one enough. Adding more infrastructure go only add more ways for things to fail. Use another queue or proxy in front when any of these conditions apply.
- You need priority. Interactive chat no suppose wait behind batch summarisation job. Ollama queue no get priority, so you gats hold batch work outside and dey feed am small small.
- You need fairness. One client fit occupy every slot and pile up waiting requests by itself. Everybody else go then wait behind that client, or get 503 if their request need model wey no dey loaded.
- You need a bounded wait. Ollama go hold requests for model wey don already load indefinitely. So, the only reliable limit na the one wey you put in front.
- You need the work to survive restart. The queue dey live inside server memory. If you restart Ollama, every waiting request go disappear.
- You need real retries with backoff, recorded for place wey you fit inspect later.
The lighter option na reverse proxy. For nginx, limit_conn go limit simultaneous connections, while limit_req go limit arrival rate per client. This mean say proxy go refuse overflow, and e no go reach Ollama. The heavier option na job queue with database in front of worker wey dey call Ollama. Use this one when requests must survive process restart. Sizing am for real traffic na separate work: planning a self hosted LLM for concurrent users explain the arithmetic, while running Ollama on a VPS cover the basic installation wey these variables assume.
Di honest answer fit be another server
Get limit wey tuning no fit pass. Ollama dey split KV cache into equal, fixed slots when model load. Memory for idle slot no fit serve busy slot, and slot count no fit change unless you unload model. This design good for one person, small team, or coding agent.
Servers wey many users dey use at the same time dey work differently. Dem dey allocate KV cache in small pages as demand come, then add new requests to batch wey already dey run. So memory dey follow real demand instead of fixed division. If your goal na plenty concurrent users for one GPU, this architecture difference matter pass any value of OLLAMA_NUM_PARALLEL. The comparison between Ollama and vLLM na where you suppose make this decision. But no switch just because of principle. Another server dey require more operations work, and if na only few people dey use your system, the built-in behaviour na the correct answer. If the real question na whether you suppose run model server by yourself at all, comparing a quote-only private AI server with building the same stack on a rented GPU go explain the decision, including when API na the better option.
Measure your own throughput and time to first token
Tokens per second figures wey people publish come from another person GPU, model, quantisation, context length, and prompt. None of these go match your own setup, so treat any number wey you read as rough hint and measure the machine wey dey in front of you.
Ollama dey return timing for the final JSON object of every response. eval_count na the number of tokens wey e generate, and eval_duration na the time wey e spend to generate dem, for nanoseconds.
sudo apt install -y jq
curl -s http://127.0.0.1:11434/api/generate \
-d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":false}' \
| jq '{prompt_eval_count, eval_count, eval_duration, tokens_per_second: (.eval_count / (.eval_duration / 1000000000))}'Run am with one slot first. Then run am again with the concurrency wey you actually expect. Compare the two numbers wey decide whether users go happy: time to first token, and tokens per second per request. Throughput per request dey always reduce as you add slots. The question be whether e go reduce pass wetin your users fit accept. Measuring tokens per second for local LLM explain the method in more detail, including how to keep the prompt constant between runs.
Public endpoint wey get plenty queue na denial of service target
Setting OLLAMA_HOST=0.0.0.0:11434 go put the API for every interface, and Ollama no get authentication wey e build inside. With the default queue, open endpoint go accept up to 512 requests wey dey wait for model to load, from anybody wey discover am. If model don already load, e go hold every request wey connect, without limit, for as long as each client remain connected. To tie server down like this almost cost attacker nothing: long prompts, no login, no rate limit, no bill. Your own users go then get 503 responses or wait for long time, and the machine go remain busy throughout.
Keep the listener for loopback and reach am through SSH tunnel or private network, or put authentication and rate limiting in front of am. How to secure Ollama API endpoint cover both options. Tune the queue after you don do that, because queue length na capacity setting, and e no protect anything.
FAQ
Why my second Ollama request dey wait make the first one finish?
Because OLLAMA_NUM_PARALLEL default na 1, so loaded model dey run one request at a time and the rest dey wait for order. The request wey dey wait keep its HTTP connection open and e no send any bytes until slot free, so e dey look like slow model from client side. The sign na the timing pattern: long pause, then text dey come full speed na queue; but slow trickle from first token na slow model. Increase the slot count with systemd drop-in and restart the service. Then check the -np value for runner line, because some model architectures dey limited to one slot no matter wetin you set.
Wetin "server busy, please try again. maximum pending requests exceeded" mean?
Na Ollama queue overflow error, and e dey return with HTTP status 503. The number of requests wey dey wait for scheduling, meaning say dem dey wait for their model to load or reload, reach OLLAMA_MAX_QUEUE. The default na 512, so the newest request reject instead of adding am. Requests to model wey don already load no dey count for that limit. You go mostly see this error during startup bursts or while models dey swap. E no be memory error and e no be model error. If you raise the queue, callers go only wait longer before the same rejection happen. The real fixes na more slots if your VRAM fit handle am, less incoming load, or queue for front wey fit retry and give priority.
Raising OLLAMA_NUM_PARALLEL go make Ollama faster?
No. E allow more requests run at the same time, but each request go slower than e for run alone because dem dey share one GPU. E also multiply the KV cache, because Ollama start the runner with total context wey equal your context length times the slot count. If the result no fit inside VRAM again, Ollama go move layers to CPU and every request go slow down, including one request wey no get competition. Check ollama ps after the change and confirm say the PROCESSOR column still dey read 100% GPU.
I need restart Ollama after I change these variables?
Yes. The server dey read dem when e start, and running model dey keep the slot count wey enter its runner process when e launch. Edit the drop-in with sudo systemctl edit ollama.service, then run sudo systemctl daemon-reload and sudo systemctl restart ollama. Confirm with systemctl show ollama --property=Environment, then check the server config line inside journalctl -u ollama, wey list the environment wey the server actually load.
How many parallel slots I suppose set?
Start with 1 and increase am one step at a time. After every step, restart Ollama, send one request to load the model, and run ollama ps. Stop for the last value where PROCESSOR still dey read 100% GPU and the SIZE column still get enough headroom for the longest context wey you serve. Then measure time to first token and tokens per second for that setting under your real concurrency. If per-request speed don drop below wetin your users fit tolerate, reduce am by one step.