Ollama vs vLLM: which LLM server to run

Ollama vs vLLM, in one paragraph

Ollama is a model manager with a server attached: it downloads quantized weights, loads them, and answers on 127.0.0.1:11434, on a CPU if that is all the box has. vLLM is a throughput engine: it keeps a GPU saturated with many requests running at the same time, and it is the wrong tool on a machine without one. That is the whole decision. One person talking to a local assistant is an Ollama job. An application serving a team is a vLLM job.

Both speak an OpenAI-compatible HTTP API, so client code moves between them by changing a base URL. The API is not the difference. The difference is what happens when a second request arrives while the first one is still generating tokens.

What Ollama actually is

Ollama is a convenience layer. It gives you a model registry (ollama pull llama3.1:8b), a local store of weights, a chat prompt, a systemd service, and an HTTP API, from one install command. The models it serves are GGUF files, usually 4-bit quantized, which is why a 7B or 8B model is around 5 GB on disk instead of 16 GB. Quantization is what makes CPU inference possible at all.

Its runner is built on llama.cpp, the C++ inference library that made GGUF quantization practical on ordinary hardware. Ollama has since added its own engine for some newer model families, but llama.cpp is still the substrate under most of what it serves. So when people compare Ollama with llama.cpp, they are mostly comparing an ergonomics layer with the thing it wraps.

The design target is one user. As of July 2026 the default for OLLAMA_NUM_PARALLEL is 1, which means one model processes one request at a time and everything else waits in a queue that holds 512 entries by default (OLLAMA_MAX_QUEUE). You can raise the parallel setting, and the section below explains what it costs you. If you have not run Ollama before, start with hosting Ollama on a VPS and keeping port 11434 closed, because the API has no authentication of any kind.

What vLLM actually is

vLLM is an inference server and nothing else. It does not manage a model library, it has no chat prompt, and it will not pull a model for you at request time. You name a Hugging Face repository at launch, it loads that one model, and it serves it until you stop the process.

What you get for that narrowness is throughput. Two mechanisms do the work. PagedAttention stores the KV cache (key-value cache, the per-token attention state a model keeps for every active request) in fixed-size blocks, the way an operating system pages memory. A request no longer needs one large contiguous reservation sized for the worst case, so memory that used to sit reserved and unused becomes available for more concurrent requests. Continuous batching lets a new request join the running batch at the next decoding step instead of waiting for the current batch to finish. A finished sequence leaves the batch immediately and its slot is refilled.

The practical result: on a single GPU, going from one concurrent user to thirty raises total tokens per second sharply, while per-user speed falls much less than you would expect. Under Ollama's default, going from one user to thirty just makes twenty-nine people wait.

Continuous batching is the whole difference

Picture five requests hitting each server at the same moment on identical hardware.

Ollama with default settings runs request one to completion, then request two, and so on. The fifth caller waits for four full generations. Total throughput is roughly the speed of one generation, because the processor is only ever working on one sequence.

vLLM decodes all five in the same forward pass. Generating one token for five sequences costs barely more than generating one token for one sequence, because the expensive part is reading the model weights out of memory, and that read is shared across the whole batch. This is the same memory-bandwidth fact that makes CPU inference slow: you pay for moving weights, not for the arithmetic.

You can set OLLAMA_NUM_PARALLEL=4 and get some of this. The cost is memory. Each parallel slot needs its own KV cache, and Ollama divides the context window across the slots, so four parallel requests against a model configured for 8192 tokens leave each request 2048 tokens of context. vLLM's paged cache is what avoids that trade, because blocks are allocated to a request as the request actually grows.

Install and serve with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run --verbose llama3.1:8b "Write two sentences about Linux."

The install script creates an ollama system user, installs the binary, and registers ollama.service bound to 127.0.0.1:11434. The eval rate line printed by --verbose is your real tokens per second on that box. Trust it over any published figure.

To raise concurrency, use a systemd drop-in so an upgrade does not overwrite the change:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl restart ollama
ollama ps

ollama ps shows what is loaded, and its PROCESSOR column tells the truth. 100% CPU means no GPU is involved, which is the honest explanation for most reports that Ollama is slow.

Install and serve with vLLM

vLLM needs Linux and Python 3.10 to 3.13. Install it into its own virtual environment, because it pulls in a specific PyTorch build:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Then serve a model. The name is a Hugging Face repository id, not a short tag:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Startup is slow the first time, because it downloads the weights and then profiles the GPU to decide how many KV cache blocks fit. It listens on port 8000. Check it before writing any client code:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]}'

If Docker is already on the box, the official image avoids the CUDA dependency work:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

--ipc=host is required, not decorative: PyTorch passes tensors between processes through shared memory, and the default Docker shared-memory allocation is too small for tensor-parallel inference.

The flags that matter most in production are --max-model-len (the context window you are willing to pay for), --gpu-memory-utilization (the fraction of the card vLLM may claim, 0.92 by default as of July 2026), --tensor-parallel-size for splitting one model across several GPUs, and --api-key.

Authentication is one flag on vLLM and absent on Ollama

vLLM enforces a bearer token if you give it one:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key token-abc123

The same value can come from the VLLM_API_KEY environment variable. A request without it gets an HTTP 401. That is still not a reason to publish port 8000 on a public interface, since vLLM has no rate limiting and a plain HTTP token is readable in transit, but it does mean the server has a concept of a caller.

Ollama has none. There is no key, no login, no allow-list. Any process that can reach 11434 can run, pull, or delete models. Keep it on loopback and reach it over a WireGuard VPN you host yourself, or through an authenticating reverse proxy that terminates TLS (transport layer security).

Hardware: what each one needs

Ollama runs on CPU. A 4-bit quantized model costs roughly half a gigabyte of RAM per billion parameters, plus about a gigabyte of runtime overhead and more for context, so a 3B model wants around 4 GB free and an 8B model around 8 GB. Speed on a shared vCPU is single digit to low double digit tokens per second. That is memory bandwidth, not a misconfiguration, and no flag repairs it.

vLLM assumes a GPU. Its default path serves unquantized weights at 16-bit precision, which is roughly 2 GB per billion parameters: an 8B model needs about 16 GB of video memory for weights alone, before the KV cache that gives you the concurrency you installed vLLM for. On a 24 GB card that leaves a workable cache. On a 16 GB card it does not, so you either pick a smaller model or pass --quantization with a quantized checkpoint. A CPU backend exists, but the standard wheels are not built for it, and it discards the reason to run vLLM at all.

The hardware question therefore answers the software question most of the time. No GPU means Ollama. A rented GPU sitting at 5 percent utilisation because requests are being serialised means vLLM.

Which one for your workload

• One person, a CPU VPS, drafting and summarising: Ollama. The pace is acceptable and nothing else is simpler.
• A coding assistant, or an MCP server bridging your tools to a local model, that only you call: Ollama. Concurrency of one is the actual workload.
• Comparing five models this week: Ollama. Pulling and deleting tagged models is exactly what it is good at, while vLLM needs a process restart per model.
• An internal app, a chat product, or a retrieval pipeline with real users: vLLM. This is where batching earns a GPU bill.
• A batch job scoring a hundred thousand documents overnight: vLLM, with a high --max-num-seqs. Throughput is the only metric that matters and per-document latency does not.
• An agent platform where several self-hosted AI agents hit the model at once: vLLM, because agent traffic is bursty and parallel by nature.

Failure modes, with the strings you will see

vLLM refuses to start with a KV cache error. The message names both numbers:

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (8192). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

The model declares a context window larger than the memory left after loading the weights. Lower it with --max-model-len 8192, or raise --gpu-memory-utilization if nothing else is using the card. Pushing utilisation past about 0.95 tends to trade this startup error for a CUDA out-of-memory crash later, under load, which is the worse of the two.

Ollama prints Killed mid-generation. The Linux out-of-memory killer stopped the process because the model needed more RAM than the box has. Confirm with sudo dmesg | grep -i oom. The fix is a smaller or more heavily quantized model, not a setting.

Ollama answers fine alone and stalls under load. No error appears anywhere. Requests simply take longer the more callers there are, because OLLAMA_NUM_PARALLEL=1 is serialising them. Raise it and accept the smaller per-request context, or move the workload to vLLM.

vLLM returns 401 on every call. You started it with --api-key and the client sends no Authorization header. Most OpenAI client libraries send whatever you pass as the key, so set it there rather than dropping the flag.

vLLM says the model is not found. Ollama pulls on demand, vLLM does not. The model field in the request body must match the repository id you launched with, or the value of --served-model-name if you set one. Confirm the exact string with curl http://localhost:8000/v1/models.

Running both is a reasonable answer

They are not exclusive. A common shape is vLLM on a GPU instance serving the application, with Ollama on the ordinary VPS beside it for local scripts, cron jobs, and trying new model releases. Both endpoints are OpenAI-compatible, so one client library and a base-URL switch cover them. Cost control matters more here than either engine does, because an idle GPU bills the same as a busy one, and keeping agent and inference costs predictable is a separate discipline from picking a server.

FAQ

Is vLLM faster than Ollama?

For a single request on the same GPU the gap is modest, since both are doing the same arithmetic. For many concurrent requests vLLM is far ahead, because continuous batching decodes every active sequence in one forward pass while Ollama's default runs them one after another. On a CPU-only machine the question does not apply: Ollama runs there and vLLM effectively does not.

Can vLLM run without a GPU?

Not usefully. The standard wheels target NVIDIA or AMD GPUs, and the reason vLLM exists, keeping an accelerator saturated with batched requests, disappears on a CPU. A CPU backend exists for development work. For real CPU inference use Ollama or llama.cpp directly.

What is the difference between Ollama and llama.cpp?

llama.cpp is the inference library, and GGUF is its quantized weight format. Ollama's runner is built on it and adds the parts llama.cpp leaves to you: a model registry, automatic download, a resident server, a systemd unit, and an OpenAI-compatible endpoint. Ollama has added its own engine for some newer model families, so the two are no longer identical underneath.

How much GPU memory does vLLM need for an 8B model?

At 16-bit precision the weights alone are about 16 GB, roughly 2 GB per billion parameters, and the KV cache needs room on top of that. A 24 GB card is comfortable. A 16 GB card requires a quantized checkpoint or a smaller model. vLLM claims a fraction of the card set by --gpu-memory-utilization, which defaults to 0.92 as of July 2026.

Do I need to change my application code to switch between them?

Usually only the base URL, the API key, and the model name. Ollama serves its OpenAI-compatible surface at http://127.0.0.1:11434/v1 and ignores the key, while vLLM serves http://localhost:8000/v1 and enforces the key if you set one. The model names differ in form: llama3.1:8b for Ollama, a full repository id such as Qwen/Qwen2.5-1.5B-Instruct for vLLM.