Ollama vs vLLM: Which One Fit Run Your LLM?

Ollama vs vLLM, for one paragraph

Ollama na model manager wey get server join am: e dey download quantized weights, load dem, then answer for 127.0.0.1:11434, for CPU if na only thing wey the box get. vLLM na throughput engine: e dey keep GPU busy with plenty requests wey dey run at the same time, and e no be the correct tool for machine wey no get GPU. Na so the decision simple. One person wey dey talk to local assistant na Ollama job. Application wey dey serve one team na vLLM job.

Both of dem dey speak OpenAI-compatible HTTP API, so client code fit move between dem if you change base URL. API no be the difference. The difference na wetin dey happen when second request arrive while the first one still dey generate tokens.

Wetin Ollama really be

Ollama na convenience layer. E give you model registry (ollama pull llama3.1:8b), local store for weights, chat prompt, systemd service, and HTTP API, all from one install command. Most models wey e serve na GGUF files, usually 4-bit quantized. Na why 7B or 8B model fit occupy around 5 GB for disk instead of 16 GB. Quantization reduce memory wey CPU inference need. For Apple Silicon Macs, e fit also run safetensors-based models through MLX engine. Since v0.40.0 (September 2026), supported architectures dey run on MLX automatically.

Ollama GGUF runner dey built on llama.cpp, the C++ inference library wey make GGUF quantization practical for ordinary hardware. As of October 2026, Ollama support two engines: llama.cpp and MLX. Linux builds also support MLX inference with CUDA. Development guide enable this backend with cmake -B build . -DOLLAMA_MLX_BACKENDS=cuda_v13, with CUDA 13+ and cuDNN 9+ as prerequisites. So when people compare Ollama with llama.cpp, mostly dem dey compare ergonomics layer with the thing wey e wrap.

The design target na one user. As of October 2026, default value for OLLAMA_NUM_PARALLEL na 1. This mean one model dey process one request at a time, while everything else dey wait for queue wey hold 512 entries by default (OLLAMA_MAX_QUEUE). You fit increase parallel setting, and the section below explain the cost. If you never run Ollama before, start with hosting Ollama for VPS and keeping port 11434 closed, because local API no require authentication.

Wetin vLLM really be

vLLM na inference server and nothing else. E no manage model library, e provide vllm chat client for server wey dey run, and e no go pull model for you when request come. You go specify Hugging Face repository when you launch am, e go load that one model, and e go serve am until you stop the process.

Wetin you gain from this narrow scope na throughput. Two mechanisms dey make am work. PagedAttention dey store KV cache (key-value cache, na the attention state wey model dey keep for every active request, token by token) inside fixed-size blocks, the same way operating system dey page memory. Request no longer need one large contiguous reservation wey dem size for worst case, so memory wey bin dey reserved but unused fit become available for more concurrent requests. Continuous batching let new request join the batch wey dey run for the next decoding step, instead of waiting for current batch to finish. When sequence finish, e comot from batch immediately and another request fill the slot.

The practical result be say: for one GPU, when concurrent users increase from one to thirty, total tokens per second go rise sharply, while each user's speed go reduce far less than you expect. With Ollama default, increasing users from one to thirty just make twenty-nine people wait.

Continuous batching na the whole difference

Imagine say five requests hit each server for the same time, with identical hardware.

With default settings, Ollama run request one finish first, then request two, and so on. The fifth caller go wait for four complete generations. Total throughput dey roughly equal to the speed of one generation, because the processor dey work on only one sequence at any time.

vLLM decode all five inside the same forward pass. To generate one token for five sequences cost barely pass generating one token for one sequence, because the expensive part na reading the model weights from memory, and that read dey shared across the whole batch. Na this same memory-bandwidth fact dey make CPU inference slow: you dey pay for moving weights, not for the arithmetic.

You fit set OLLAMA_NUM_PARALLEL=4 and get some of this behaviour. The cost na memory. Every parallel slot need its own KV cache, and Ollama multiply the context allocation instead of dividing am: its FAQ talk say required RAM dey scale by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. So four parallel requests against a model configured for 8192 tokens go keep 8192 tokens each, but the box must hold cache for 32768 tokens altogether. That 8192 na itself a choice, e no be fixed value, so raising num_ctx and sizing the RAM wey e cost na the step wey decide whether four slots fit at all. vLLM paged cache na wetin avoid reserving the full window for every slot from the beginning, because blocks dey allocate to a request as the request dey grow. Either way, the limit on how many people one box fit serve at once depend on KV cache size, prefill cost, and queue depth, wey na why server wey feel okay for one person dey crawl at five.

Install and serve with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run --verbose llama3.1:8b "Write two sentences about Linux."

The install script dey create an ollama system user, install the binary, and register ollama.service wey dey bound to 127.0.0.1:11434. The eval rate line wey --verbose print na your real tokens per second for that machine. Trust am pass any published figure. One reading from one prompt na starting point, no be capacity number. So measure tokens per second across different concurrency levels na wetin go show whether the machine fit handle the load wey you really expect, and whether renting GPU go cost less than paying per token.

To increase concurrency, use systemd drop-in so upgrade no go overwrite the change:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl restart ollama
ollama ps

ollama ps dey show wetin load, and e PROCESSOR column dey tell the truth. 100% CPU mean say no GPU dey involved, and na the honest reason for most reports say Ollama slow. The OLLAMA_KEEP_ALIVE=30m line for that drop-in matter just as much for quiet machine, because the default go unload the model after five minutes without request. Keeping the model resident between requests na wetin stop the first prompt after a break of up to half an hour from paying the full load time again. If your quiet periods dey last pass that, set longer value.

Install and serve with vLLM

vLLM GPU builds need Linux. For macOS on Apple Silicon, na CPU-only support with prebuilt wheel for the 0.32.0 release. GPU acceleration for those Macs dey come from vLLM-Metal, wey be separate community-maintained plugin built on MLX. For Python, current release (0.32.0, October 2026) accepts 3.11 to 3.14, and the install guide list the same range. Dem drop Python 3.10 for 0.32.0, so 3.12, wey we use below, na safe choice. Install am inside im own virtual environment, because e dey install specific PyTorch build:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

The last line dey inside vLLM NVIDIA install guide as of October 2026. E dey install FlashInfer precompiled kernels. Without dem, vLLM go download and compile kernels when e start, and the guide talk say this fit add several minutes for Hopper and newer GPUs. Run am again after every vLLM upgrade, because FlashInfer no fit import when the kernels no match.

Then serve one model. The name na Hugging Face repository id, no be short tag:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The first startup slow because e dey download the weights, then e profile the GPU to decide how many KV cache blocks fit. E dey listen on port 8000. Check am before you write any client code:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]}'

If Docker already dey for the server, the official image go avoid the CUDA dependency work:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

--ipc=host give the container access to shared memory for the host. Increasing --shm-size na another option. PyTorch dey pass tensors between processes through shared memory, especially for tensor-parallel inference.

The flags wey matter pass for production na --max-model-len, wey be the context window wey you agree to pay for; --gpu-memory-utilization, wey be the fraction of the GPU card wey vLLM fit use, 0.92 by default as of October 2026; --tensor-parallel-size for splitting one model across many GPUs; and --api-key.

Authentication na just one flag for vLLM, but e no dey for Ollama local API

vLLM go enforce bearer token if you give am:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key token-abc123

The same value fit come from the VLLM_API_KEY environment variable. Request wey go the /v1 routes without am go get HTTP 401. But the key no protect the whole server. vLLM security guide, together with authentication code for the 0.32.0 source, apply am only to endpoints under the /v1, /v2, /inference, and /cohere path prefixes. The --api-key help text for the vllm serve page list only the first three, but the code check all four. Both the guide and help text talk say other endpoints, especially /invocations, wey expose the same inference as /v1, no get authentication. This alone don reach to avoid publishing port 8000 on public interface. vLLM security guide also leave rate limiting to reverse proxy wey dey in front of am. Token wey pass through plain HTTP fit dey readable during transit, but the flag still mean say server get concept of caller.

Ollama local API no require authentication. Cloud models, model publishing, and private model downloads require sign-in; direct API access to ollama.com require API key. Any process wey fit reach 11434 fit run, pull, or delete models. Keep am on loopback and reach am through WireGuard VPN wey you host yourself, or through authenticating reverse proxy wey terminate TLS (transport layer security).

Hardware: wetin each one need

Ollama dey run for CPU. 4-bit quantized model dey use roughly half gigabyte RAM for every one billion parameters, plus about one gigabyte runtime overhead and extra memory for context. So, 3B model need around 4 GB free memory, while 8B model need around 8 GB. Speed for shared vCPU na single digit to low double digit tokens per second. Na memory bandwidth cause this, no be misconfiguration, and no flag fit repair am. To see this calculation for one specific release instead of using general estimate, running Nemotron 3.5 Lightning for VPS dey specify the exact tag to pull, the RAM wey e dey hold after loading, and whether CPU-only speed dey quick enough to use.

vLLM assume say GPU dey available. E default path dey serve unquantized weights for 16-bit precision. This na roughly 2 GB for every one billion parameters. So, 8B model need about 16 GB video memory for weights alone, before the KV cache wey give you the concurrency wey make you install vLLM. For 24 GB card, enough workable space dey remain for cache. For 16 GB card, e no go work. You go need choose smaller model, pass --quantization (with quantized checkpoint, or make e quantize full-precision weights during load time), or move part of the weights enter system RAM with --cpu-offload-gb. This one preserve their precision but e dey slower, because those weights cross go GPU for every forward pass. CPU backend dey available, with separate wheels, but e remove the main reason to run vLLM.

So, hardware question dey answer software question most times. If GPU no dey, use Ollama. If rented GPU dey use only 5 percent because requests dey run one after another, use vLLM.

Which one fit your workload

• One person, one CPU VPS, dey draft and summarise: Ollama. The speed dey acceptable and nothing else simple pass am.
• Coding assistant, or MCP server wey dey connect your tools to local model, wey na only you dey call: Ollama. Concurrency of one na the real workload.
• If you dey compare five models this week: Ollama. Pulling and deleting tagged models na exactly wetin e good for, while vLLM need process restart for each model.
• Internal app, chat product, or retrieval pipeline wey get real users: vLLM. Na here batching justify GPU bill.
• Batch job wey dey score hundred thousand documents overnight: vLLM, with high --max-num-seqs. Throughput na the only metric wey matter, and per-document latency no matter.
• Agent platform wey several self-hosted AI agents dey use to reach the model at the same time: vLLM, because agent traffic dey bursty and e naturally dey happen in parallel.

Failure modes, with the strings you will see

vLLM no gree start and e show KV cache error. The message go name both numbers:

ValueError: To serve at least one request with the model's max seq len (32768), (4.00 GiB KV cache is needed, which is larger than the available KV cache memory (1.00 GiB). Based on the available memory, the estimated maximum model length is 8192. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.

The model declare context window wey big pass the memory wey remain after e load the weights. Lower am with --max-model-len 8192, or increase --gpu-memory-utilization if nothing else dey use the card.

Ollama print Killed for middle of generation. Linux out-of-memory killer stop the process because the model need more RAM pass wetin the box get. Confirm am with sudo dmesg | grep -i oom. The fix na smaller or more heavily quantized model, no be setting.

Ollama answer well when e dey alone but e stall under load. No error dey show anywhere. Requests just dey take longer as callers increase, because OLLAMA_NUM_PARALLEL=1 dey serialize dem. Long answers make the queue worse, because one caller wey hold the only slot until the model decide to stop go block everybody behind am, so limit the reply with num_predict go set maximum time wey any single turn fit hold the server. Increase the parallel setting and accept the extra RAM wey each slot context cost, or move the workload go vLLM.

vLLM return 401 for every /v1 call. You start am with --api-key and the client no send Authorization header. Most OpenAI client libraries send anything wey you pass as the key, so set am there instead of removing the flag.

vLLM talk say the model no dey found. Ollama dey pull on demand, but vLLM no dey do am. The model field for the request body must match the repository id wey you launch with, or the value of --served-model-name if you set one. Confirm the exact string with curl http://localhost:8000/v1/models.

Run both na reasonable answer

Dem no dey exclude each other. One common setup na to run vLLM for GPU instance to serve the application, while Ollama dey run for the ordinary VPS beside am for local scripts, cron jobs, and to try new model releases. Both endpoints dey OpenAI-compatible, so one client library and base-URL switch fit handle both. Cost control matter pass either engine for this case, because idle GPU still dey bill like busy one, and keeping agent and inference costs predictable na separate discipline from choosing server.

FAQ

vLLM dey faster pass Ollama?

For one request on the same GPU, difference no too big, because both dey do the same arithmetic. For plenty concurrent requests, vLLM dey far ahead because continuous batching dey decode every active sequence for one forward pass, while Ollama default dey run dem one after another. For machine wey na CPU-only, both get supported inference paths, but vLLM dey use separate CPU wheels.

vLLM fit run without GPU?

Yes, through im CPU backend. Standard wheels target NVIDIA or AMD GPUs, and the reason wey vLLM dey exist — to keep accelerator busy with batched requests — no really apply for CPU. CPU backend dey available, with separate wheels for x86 and Arm. For real CPU inference, use Ollama or llama.cpp directly.

Wetin be the difference between Ollama and llama.cpp?

llama.cpp na the inference library, while GGUF na im quantized weight format. Ollama dey use am for GGUF inference and e add model registry, tagged downloads, model management, and systemd unit. Ollama still dey ship MLX engine wey dey run safetensors-based models for Apple Silicon and support CUDA builds for Linux, so the two no always be the same underneath.

How much GPU memory vLLM need for 8B model?

For 16-bit precision, the weights alone na about 16 GB, roughly 2 GB for every billion parameters, and KV cache still need extra space. 24 GB card dey comfortable. 16 GB card need quantized model, smaller model, or --cpu-offload-gb to move part of the weights enter system RAM, but speed go reduce. vLLM dey reserve a fraction of the card set by --gpu-memory-utilization, wey default to 0.92 as of October 2026.

I need change my application code to switch between dem?

Usually, na only the base URL, API key, and model name. Ollama dey serve im OpenAI-compatible surface for http://127.0.0.1:11434/v1 and e ignore the key, while vLLM dey serve http://localhost:8000/v1 and enforce the key for im /v1 routes if you set one. The model names get different format: llama3.1:8b for Ollama, and full repository id like Qwen/Qwen2.5-1.5B-Instruct for vLLM.