Ollama concurrency: NUM_PARALLEL and MAX_QUEUE

What happens to the second Ollama request while the first is generating

Ollama concurrency is decided by three environment variables, and out of the box one loaded model serves one request at a time. The second request is not refused, and it does not receive a partial answer. It waits in a queue until a slot is free, and then it runs at normal speed.

An incoming request has three possible fates. It starts immediately in a free slot. It waits in the queue. Or the queue is already full and the server refuses it with HTTP 503. Which one you get is decided by OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS.

The default is safe, and it is also why a second user reports that the server "hung" when nothing is broken. Adding slots is a two line change. The part that bites is memory. Every parallel slot needs its own key/value cache (KV cache), the block of memory a model keeps for the tokens it has already processed. Add slots without adding VRAM (video memory on the GPU) and you turn a slow answer into a failed load.

What OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS each control

These are the defaults in current Ollama releases as of August 2026. Check yours rather than trusting the number here, using the log line shown further down.

• OLLAMA_NUM_PARALLEL is how many requests one loaded model handles at the same time. The default is 1, so requests are served one after another.
• OLLAMA_MAX_LOADED_MODELS is how many different models stay resident at once. The default is 0, which means Ollama picks: three models per GPU, and three on a machine with no GPU.
• OLLAMA_MAX_QUEUE is how many requests may sit waiting. The default is 512. The request that arrives when the queue is full is rejected straight away.

The worst case memory is the product of the first two. Two loaded models with four slots each is eight slot allocations of KV cache, all resident at once, and Ollama will try to satisfy that. On a single GPU box it is usually better to hold one model and give it slots, because the arithmetic stays something you can do in your head.

Why every parallel slot costs VRAM

When Ollama loads a model it launches a separate runner process. Two of the arguments it passes matter here: -c is the total context the runner allocates a KV cache for, and -np is the number of parallel sequences. Ollama sets -c to your per-request context length multiplied by the slot count. The runner then divides that total evenly between the slots, so each request still gets the context length you asked for.

That is the whole constraint, and it is why parallelism is not free. Going from one slot to four asks for four times the KV cache at the same per-request context. Nothing is shared between slots, and an idle slot's share is not lent to a busy one, because the split is fixed when the runner starts.

You can read the real numbers instead of the ones you meant to set:

journalctl -u ollama --no-pager -n 500 | grep "starting llama-server"

That line holds the full runner command line, including -c and -np. If -np is 1 after you set the variable, the setting is not reaching the server, and the next section covers why.

If the model weights plus that KV cache do not fit in VRAM, Ollama moves some layers to system RAM and those layers run on the CPU. CPU layers are far slower than GPU layers, so this makes every request slower, including the single request you started with. Raising parallelism can therefore reduce throughput rather than raise it.

ollama ps

The PROCESSOR column reads 100% GPU when the whole thing fits. A split such as 35%/65% CPU/GPU means part of the model is running on the CPU. The SIZE column includes the KV cache, so it grows when you raise the slot count and reload the model. Raise OLLAMA_NUM_PARALLEL, restart, send one request, and run ollama ps again: that is the memory cost of your change, measured rather than guessed.

Context length and slot count multiply, so they have to be chosen together. A large context with four slots is four large contexts. If you are also tuning the num_ctx context window for your model, change one of the two at a time, or you will not know which one filled the card.

How to set these variables so they survive a reboot

On Linux, Ollama runs as a systemd service. Running export OLLAMA_NUM_PARALLEL=4 in your shell changes nothing, because systemd starts the service with its own environment and never sees your shell. Use a drop-in file.

sudo systemctl edit ollama.service

Add this in the editor that opens:

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_MAX_QUEUE=32"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama --property=Environment

systemctl show prints what systemd will hand the process. If your variable is missing there, the drop-in was not saved or daemon-reload was skipped. Confirm from the server's own side too:

journalctl -u ollama --no-pager | grep "server config" | tail -1

Ollama logs its entire environment at startup on a line whose message is server config. That map is the truth. It is the fastest way to settle an argument about whether a variable took effect.

A model that is already loaded keeps the slot count it was started with, because the value is fixed into the runner process at launch. The restart above unloads everything, so the next request reloads the model with the new setting and pays the load time once. How long a model stays resident after that is a separate control, covered in keeping an Ollama model loaded between requests.

What served, queued and refused look like from the client

Send several requests at once and time them. This runs eight streaming requests in parallel and prints the status and the timings for each:

for i in $(seq 1 8); do
  curl -s -o /dev/null \
    -w "req$i http=%{http_code} ttfb=%{time_starttransfer}s total=%{time_total}s\n" \
    http://127.0.0.1:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":true}' &
done
wait

ttfb is the time to the first byte of the stream, which is close to the time to first token (TTFT) because the first streamed chunk carries the first token.

Served in parallel. Every request reports a similar ttfb, and total rises for all of them together. The GPU is shared between the running slots, so each answer is slower than it would be alone while more of them finish per minute. This is the regime you are buying when you raise OLLAMA_NUM_PARALLEL.

Queued. The first requests answer quickly and the later ones show a large ttfb followed by a normal generation. The wait is the queue, not the model. A user watching a chat window sees a long blank pause and then text at full speed. That shape, slow to start and then fast, is the signature of a queue rather than an overloaded GPU.

Refused. The client gets http=503 almost instantly, and the body is:

{"error":"server busy, please try again.  maximum pending requests exceeded"}

That message means the queue was full at the moment the request arrived. It says nothing about VRAM and nothing about the model.

One honest limit: Ollama does not publish a queue depth. ollama ps and the /api/ps endpoint report the models that are loaded, not the requests that are waiting. So you measure the queue from the client side, by watching time to first byte, or you count 503 responses at whatever sits in front.

Why a smaller MAX_QUEUE is often the better setting

A queue of 512 sounds generous, and on a single slot it is close to useless. Request 300 waits behind 299 complete generations. That is minutes at best. Every HTTP client gives up long before then, so the caller sees a client side timeout, which tells them nothing about the cause and gives your monitoring nothing to alert on.

Set the queue to roughly what your server can clear inside your client's timeout, and the overflow becomes an immediate 503 instead. A 503 is useful: a reverse proxy can retry it, a client can back off, a dashboard can count it, and a person can read it. Work the number out from your own measurements. If a generation takes about ten seconds and your client waits sixty, then about six requests per slot can clear inside that window, and a queue much deeper than that only produces timeouts.

When to put a queue in front of Ollama

The built-in queue is first in, first out (FIFO) and it knows nothing about who is calling. For one application talking to one server, that is enough, and adding infrastructure would only add failure modes. Reach for something in front when one of these is true.

• You need priority. An interactive chat should not wait behind a batch summarisation job. Ollama's queue has no priority, so batch work has to be held outside and fed in slowly.
• You need fairness. One client can fill the queue by itself, and everyone else then gets 503.
• You need the work to survive a restart. The queue lives in the server's memory. Restart Ollama and every waiting request is gone.
• You need real retries with backoff, recorded somewhere you can inspect afterwards.

The light version is a reverse proxy. In nginx, limit_conn caps simultaneous connections and limit_req caps the arrival rate per client, so the overflow is refused at the proxy and never reaches Ollama's queue. The heavy version is a job queue with a database in front of a worker that calls Ollama, which is what you want once requests must outlive a process restart. Sizing that for real traffic is its own exercise: planning a self hosted LLM for concurrent users goes through the arithmetic, and running Ollama on a VPS covers the base install these variables assume.

When the honest answer is a different server

There is a limit you cannot tune your way past. Ollama splits the KV cache into equal, fixed slots when the model loads. An idle slot's memory cannot be used by a busy one, and the slot count cannot change without unloading the model. That design is a good fit for one person, or a small team, or a coding agent.

Servers built for many simultaneous users work differently. They allocate KV cache in small pages on demand and add arriving requests into a batch that is already running, so memory follows actual demand instead of a fixed division. If your goal is a large number of concurrent users on one GPU, that architectural difference matters more than any value of OLLAMA_NUM_PARALLEL. The comparison between Ollama and vLLM is the place to make that call. Do not switch on principle, though: a different server is more to operate, and if your traffic is a few people, the built-in behaviour is the correct answer.

Measure your own throughput and time to first token

Published tokens per second figures come from someone else's GPU, model, quantisation, context length and prompt. None of those match yours, so treat any number you read as a rough hint and measure the machine in front of you.

Ollama returns timings in the final JSON object of every response. eval_count is the number of tokens generated and eval_duration is the time spent generating them, in nanoseconds.

sudo apt install -y jq
curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Explain what a KV cache is.","stream":false}' \
  | jq '{prompt_eval_count, eval_count, eval_duration, tokens_per_second: (.eval_count / (.eval_duration / 1000000000))}'

Run that with one slot, then again at the concurrency you actually expect, and compare the two numbers that decide whether users are happy: time to first token, and tokens per second per request. Throughput per request always falls as slots are added. The question is whether it falls further than your users will accept. Measuring tokens per second on a local LLM covers the method in more detail, including how to keep the prompt constant between runs.

A public endpoint with a generous queue is a denial of service target

Setting OLLAMA_HOST=0.0.0.0:11434 puts the API on every interface, and Ollama has no built-in authentication. An open endpoint with the default queue will accept 512 waiting requests from anyone who finds it. Filling that queue costs an attacker almost nothing: long prompts, no login, no rate limit, no bill. Your own users then get 503 responses or long waits, and the machine is busy the whole time.

Keep the listener on loopback and reach it through an SSH tunnel or a private network, or put authentication and rate limiting in front of it. Securing an Ollama API endpoint covers both. Tune the queue after that is done, because a queue length is a capacity setting, and it does not protect anything.

FAQ

Why does my second Ollama request wait for the first one to finish?

Because OLLAMA_NUM_PARALLEL defaults to 1, so a loaded model runs one request at a time and the rest wait in order. The waiting request holds its HTTP connection open and sends no bytes until a slot frees, which looks identical to a slow model from the client side. The tell is the shape of the timing: a long pause and then text at full speed is a queue, while a slow trickle from the first token is a slow model. Raise the slot count with a systemd drop-in and restart the service.

What does "server busy, please try again. maximum pending requests exceeded" mean?

That is Ollama's queue overflow error, returned with HTTP status 503. The number of requests already waiting reached OLLAMA_MAX_QUEUE, which defaults to 512, so the newest request was rejected instead of being added. It is not a memory error and it is not a model error. Raising the queue only makes callers wait longer before the same rejection, so the real fixes are more slots if you have the VRAM for them, less incoming load, or a queue in front that can retry and prioritise.

Does raising OLLAMA_NUM_PARALLEL make Ollama faster?

No. It lets more requests run at the same time, and each of those requests is slower than it would be running alone, because they share one GPU. It also multiplies the KV cache, since Ollama starts the runner with a total context of your context length times the slot count. If the result no longer fits in VRAM, Ollama pushes layers onto the CPU and every request slows down, including a single one with no competition. Check ollama ps after the change and confirm the PROCESSOR column still reads 100% GPU.

Do I need to restart Ollama after changing these variables?

Yes. The server reads them at startup, and a running model keeps the slot count that was fixed into its runner process when it launched. Edit the drop-in with sudo systemctl edit ollama.service, then run sudo systemctl daemon-reload and sudo systemctl restart ollama. Confirm with systemctl show ollama --property=Environment, and then check the server config line in journalctl -u ollama, which lists the environment the server actually loaded.

How many parallel slots should I set?

Start at 1 and raise it one step at a time. After each step, restart Ollama, send one request to load the model, and run ollama ps. Stop at the last value where PROCESSOR still reads 100% GPU and the SIZE column leaves headroom for the longest context you serve. Then measure time to first token and tokens per second at that setting under your real concurrency, and back off one step if per request speed has dropped below what your users tolerate.