Ollama concurrency: NUM_PARALLEL and MAX_QUEUE gist

Wetin happen to the second Ollama request while the first one dey generate

Ollama concurrency na three environment variables dey control am, and by default, one loaded model dey serve one request at a time. The second request no dey get refused, and e no dey receive partial answer. E dey wait for queue until slot dey free, then e go run at normal speed.

Incoming request fit get three possible outcomes. E fit start immediately for free slot. E fit wait for queue. Or queue fit don full, and server go refuse am with HTTP 503. Na OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS dey decide which one you go get.

The default setting safe, but na also why second user fit report say server "hang" when nothing spoil. To add slots, na two-line change. The main wahala na memory. Every parallel slot need im own key/value cache (KV cache), wey be memory block wey model dey keep for tokens wey e don process already. If you add slots without adding VRAM (video memory on the GPU), you fit change slow answer to failed load.

Wetín OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS each dey control

Na these be the defaults for current Ollama releases as of August 2026. Check your own values instead of trusting the numbers here, using the log line wey show further down.

• OLLAMA_NUM_PARALLEL na how many requests one loaded model fit handle at the same time. The default na 1, so requests dey process one after another.
• OLLAMA_MAX_LOADED_MODELS na how many different models fit remain resident at once. The default na 0, meaning say Ollama go choose: three models per GPU, and three for machine wey no get GPU.
• OLLAMA_MAX_QUEUE na how many requests fit wait for queue. The default na 512. If request enter when queue don full, system go reject am immediately.

The worst-case memory na the product of the first two values. Two loaded models wey get four slots each mean eight KV cache slot allocations wey all remain resident at once, and Ollama go try satisfy that requirement. For one GPU box, e usually better to keep one model loaded and give am slots, because you fit calculate the arithmetic for your head.

Why every parallel slot costs VRAM

When Ollama load model, e start a separate runner process. Two arguments wey e pass matter here: -c na the total context wey runner allocate KV cache for, and -np na the number of parallel sequences. Ollama set -c to your per-request context length multiplied by the slot count. Runner then divide that total evenly across the slots, so every request still get the context length wey you ask for.

Na that be the full constraint, and na why parallelism no be free. If you move from one slot to four, e go need four times the KV cache with the same per-request context. Nothing dey shared between slots, and idle slot share no dey lend to busy one, because runner fix the split when e start.

You fit read the real numbers instead of the ones wey you intend set:

journalctl -u ollama --no-pager -n 500 | grep "starting llama-server"

That line contain the full runner command line, including -c and -np. If -np na 1 after you set the variable, the setting no reach the server, and the next section explain why.

If model weights plus that KV cache no fit inside VRAM, Ollama move some layers go system RAM, and CPU go run those layers. CPU layers slow pass GPU layers by far, so every request go slow down, including the single request wey you start with. So, increasing parallelism fit reduce throughput instead of increasing am. With model wey big enough, the weights alone go decide the matter before any slot calculation start. Na why self-hosting something the size of Kimi K3 dey concern how many cards you get, instead of how many slots you set.

ollama ps

The PROCESSOR column dey show 100% GPU when everything fit. A split like 35%/65% CPU/GPU mean say part of the model dey run on CPU. The SIZE column include the KV cache, so e go grow when you increase slot count and reload the model. Increase OLLAMA_NUM_PARALLEL, restart, send one request, then run ollama ps again: this na the memory cost of your change, measured instead of guessed. If the measurement show say model no longer fit, remember say the weights na the other half of the same budget, and moving from fp16 to a q8 or q4 build often free more VRAM than the slot wey you dey try add go cost.

Context length and slot count dey multiply, so you need choose dem together. Large context with four slots na four large contexts. If you dey also tune the num_ctx context window for your model, change only one of the two at a time, otherwise you no go know which one fill the card.

How to set these variables so dem go survive reboot

For Linux, Ollama dey run as systemd service. If you run export OLLAMA_NUM_PARALLEL=4 for your shell, e no change anything, because systemd dey start the service with e own environment and e no dey see your shell. Use drop-in file.

sudo systemctl edit ollama.service

Add this inside the editor wey go open:

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_MAX_QUEUE=32"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama --property=Environment

systemctl show go print wetin systemd wan give the process. If your variable no dey there, drop-in no save or you skip daemon-reload. Confirm am from the server side too:

journalctl -u ollama --no-pager | grep "server config" | tail -1

Ollama dey log e complete environment when e start, for line wey message be server config. Na this map be the correct source. E na the fastest way to confirm whether variable take effect.

Model wey don already load go keep the slot count wey e start with, because dem don set the value inside runner process when e launch. The restart above go unload everything, so the next request go load the model again with the new setting and pay the load time once. How long the model go remain loaded after that na separate control, and how to keep Ollama model loaded between requests cover am.

Wetin served, queued and refused dey look like from client side

Send several requests at once and time dem. This one dey run eight streaming requests in parallel and print the status plus timing for each one:

for i in $(seq 1 8); do
  curl -s -o /dev/null \
    -w "req$i http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
    http://127.0.0.1:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":true}' &
done
wait

ttfb na the time before the first byte of the stream arrive. E dey close to time to first token (TTFT), because the first streamed chunk carry the first token.

Served in parallel. Every request dey report similar ttfb, and total dey rise for all of dem together. GPU dey share between the running slots, so each answer dey slower than e for be when e run alone, while more answers dey finish every minute. Na this situation you dey buy when you increase OLLAMA_NUM_PARALLEL.

Queued. The first requests dey answer quickly, while the later ones show large ttfb before normal generation start. The wait na queue, no be model. User wey dey watch chat window go see long blank pause, then text go dey come at full speed. This pattern—slow to start, then fast—na sign of queue, no be overloaded GPU.

Refused. Client dey receive http=503 almost immediately, and the body na:

{"error":"server busy, please try again.  maximum pending requests exceeded"}

That message mean say queue full when the request arrive. E no talk anything about VRAM, and e no talk anything about model.

One honest limit dey: Ollama no publish queue depth. ollama ps and /api/ps endpoint dey report models wey don load, no be requests wey dey wait. So you measure queue from client side by watching time to first byte, or you count 503 responses from anything wey dey in front.

Why smaller MAX_QUEUE often better

Queue of 512 fit sound generous, but for one slot e almost useless. Request 300 go wait behind 299 complete generations. That one fit take minutes at best. Every HTTP client go give up long before then, so caller go see client-side timeout. This no tell dem anything about the cause, and e no give your monitoring anything to alert on.

Set the queue to roughly the number wey your server fit clear inside your client's timeout. Then overflow go become immediate 503 instead. 503 dey useful: reverse proxy fit retry am, client fit back off, dashboard fit count am, and person fit read am. Work out the number from your own measurements. If one generation dey take about ten seconds and your client dey wait sixty, about six requests per slot fit clear inside that window. Queue wey pass that by plenty go only produce timeouts.

Wetin time you suppose put queue for front of Ollama

The built-in queue dey use first in, first out (FIFO), and e no know who dey call am. If na one application dey talk to one server, this one don do. Adding more infrastructure go only add more ways for failure. Use another thing for front when any of these conditions dey apply.

• You need priority. Interactive chat no suppose wait behind batch summarisation job. Ollama queue no get priority, so you need hold batch work outside and feed am small small.
• You need fairness. One client fit fill the queue by itself, then everybody else go get 503.
• You need the work to survive restart. The queue dey inside server memory. If you restart Ollama, every request wey dey wait go disappear.
• You need proper retries with backoff, and you need record dem somewhere wey you fit inspect later.

The simple option na reverse proxy. For nginx, limit_conn dey limit simultaneous connections, while limit_req dey limit arrival rate per client. This one make proxy reject overflow, so e no ever reach Ollama queue. The bigger option na job queue with database for front of worker wey dey call Ollama. Na this one you need when requests must survive process restart. Sizing am for real traffic na separate work: planning self-hosted LLM for concurrent users dey explain the calculation, while running Ollama for VPS cover the base install wey these variables assume.

Wetin happen when another server na the honest answer

There be one limit wey tuning no fit pass. Ollama dey split KV cache into equal, fixed slots when model load. Memory for idle slot no fit serve busy slot, and slot count no fit change unless model unload. This design dey work well for one person, small team, or coding agent.

Servers wey dem build for many users wey dey use am at the same time dey work differently. Dem allocate KV cache in small pages only when request need am, and add new requests to batch wey already dey run. So memory dey follow real demand instead of fixed division. If your goal na plenty concurrent users for one GPU, this architecture difference matter pass any value of OLLAMA_NUM_PARALLEL. The comparison between Ollama and vLLM na where you go make that decision. But no switch just because of principle: another server dey require more operation work, and if na only few people dey send traffic, the built-in behaviour na the correct answer.

Measure your own throughput and time to first token

Tokens per second figures wey people publish come from another person GPU, model, quantisation, context length, and prompt. None of these go match your own setup, so treat any number wey you read as rough hint and measure the machine wey dey in front of you.

Ollama dey return timing for the final JSON object of every response. eval_count na the number of tokens wey e generate, and eval_duration na the time wey e spend to generate dem, for nanoseconds.

sudo apt install -y jq
curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":false}' \
  | jq '{prompt_eval_count, eval_count, eval_duration, tokens_per_second: (.eval_count / (.eval_duration / 1000000000))}'

Run am with one slot first. Then run am again with the concurrency wey you actually expect. Compare the two numbers wey decide whether users go happy: time to first token, and tokens per second per request. Throughput per request dey always reduce as you add slots. The question be whether e go reduce pass wetin your users fit accept. Measuring tokens per second for local LLM explain the method in more detail, including how to keep the prompt constant between runs.

Public endpoint wey get plenty queue na denial of service target

Setting OLLAMA_HOST=0.0.0.0:11434 go put the API for every interface, and Ollama no get built-in authentication. Open endpoint wey use default queue go accept 512 requests wey dey wait from anybody wey find am. E almost cost attacker nothing to fill that queue: long prompts, no login, no rate limit, no bill. Your own users go then receive 503 responses or wait for long time, while the machine dey busy throughout.

Keep the listener for loopback and reach am through SSH tunnel or private network, or put authentication and rate limiting in front of am. How to secure Ollama API endpoint cover both options. Tune the queue only after you don do that, because queue length na capacity setting, and e no protect anything.

FAQ

Why my second Ollama request dey wait make the first one finish?

Because OLLAMA_NUM_PARALLEL default na 1, so loaded model dey handle one request at a time and the rest dey wait for order. The request wey dey wait keep im HTTP connection open but e no send any byte until slot free, so for client side e look like say model slow. The timing pattern go show you: long pause, then text come out at full speed mean na queue; slow trickle from the first token mean na slow model. Increase the slot count with systemd drop-in, then restart the service.

Wetin "server busy, please try again. maximum pending requests exceeded" mean?

Na Ollama queue overflow error be that, and e dey return with HTTP status 503. The number of requests wey don already wait reach OLLAMA_MAX_QUEUE, wey default na 512, so the newest request get rejected instead of joining the queue. E no be memory error and e no be model error. If you increase the queue, callers go only wait longer before dem receive the same rejection. The real fixes na more slots if your VRAM fit handle am, less incoming load, or queue wey dey front and fit retry and prioritise requests.

OLLAMA_NUM_PARALLEL increase go make Ollama faster?

No. E go allow more requests run at the same time, but each request go slower than if e run alone because dem dey share one GPU. E also multiply the KV cache, because Ollama start the runner with total context wey be your context length times the slot count. If the result no fit inside VRAM again, Ollama go push some layers go CPU, and every request go slow down, including one request wey no get competition. Check ollama ps after the change and confirm say the PROCESSOR column still dey read 100% GPU.

I need restart Ollama after I change these variables?

Yes. The server dey read dem when e start, and running model dey keep the slot count wey e fix for im runner process when e launch. Edit the drop-in with sudo systemctl edit ollama.service, then run sudo systemctl daemon-reload and sudo systemctl restart ollama. Confirm with systemctl show ollama --property=Environment, then check the server config line inside journalctl -u ollama. That line list the environment wey the server actually load.

How many parallel slots I suppose set?

Start with 1 and increase am one step at a time. After each step, restart Ollama, send one request to load the model, then run ollama ps. Stop for the last value where PROCESSOR still dey read 100% GPU and the SIZE column still get enough headroom for the longest context wey you dey serve. Then measure time to first token and tokens per second for that setting under your real concurrency. If per-request speed drop below wetin your users fit tolerate, reduce am by one step.