Ollama vs vLLM: Which One Fit Run Your LLM?

Ollama vs vLLM, for one paragraph

Ollama na model manager wey get server join am: e dey download quantized weights, load dem, then answer for 127.0.0.1:11434, for CPU if na only thing wey the box get. vLLM na throughput engine: e dey keep GPU busy with plenty requests wey dey run at the same time, and e no be the correct tool for machine wey no get GPU. Na so the decision simple. One person wey dey talk to local assistant na Ollama job. Application wey dey serve one team na vLLM job.

Both of dem dey speak OpenAI-compatible HTTP API, so client code fit move between dem if you change base URL. API no be the difference. The difference na wetin dey happen when second request arrive while the first one still dey generate tokens.

Wetin Ollama really be

Ollama na convenience layer. E give you model registry (ollama pull llama3.1:8b), local store for weights, chat prompt, systemd service, and HTTP API, all from one install command. The models wey e serve na GGUF files, usually 4-bit quantized. Na why 7B or 8B model dey use around 5 GB for disk instead of 16 GB. Quantization na wetin make CPU inference possible at all.

Ollama runner dey build on llama.cpp, the C++ inference library wey make GGUF quantization practical for ordinary hardware. Ollama don later add its own engine for some newer model families, but llama.cpp still be the substrate under most things wey e serve. So when people compare Ollama with llama.cpp, dem mostly dey compare ergonomics layer with the thing wey e wrap.

The design target na one user. As of July 2026, the default for OLLAMA_NUM_PARALLEL na 1. This mean say one model dey process one request at a time, while everything else dey wait for queue wey hold 512 entries by default (OLLAMA_MAX_QUEUE). You fit increase the parallel setting, and the section below explain wetin e go cost you. If you never run Ollama before, start with hosting Ollama for VPS and keeping port 11434 closed, because the API no get any authentication at all.

vLLM actually be wetin

vLLM na inference server and nothing more. E no manage model library, e no get chat prompt, and e no go pull model for you when request come. You go name one Hugging Face repository when you launch am, e go load that one model, and e go serve am until you stop the process.

Wetin you gain from this narrow focus na throughput. Two mechanisms dey handle the work. PagedAttention dey store the KV cache (key-value cache, na the attention state wey model dey keep for every active request) inside fixed-size blocks, like how operating system dey page memory. Request no longer need one large contiguous reservation wey dem size for worst case, so memory wey used to remain reserved and unused fit become available for more concurrent requests. Continuous batching let new request join the running batch for the next decoding step instead of waiting for current batch to finish. When sequence finish, e leave the batch immediately and dem refill the slot.

The practical result be say: for one GPU, when you move from one concurrent user to thirty, total tokens per second go rise sharply, while speed for each user go fall much less than you fit expect. Under Ollama default, moving from one user to thirty just make twenty-nine people wait.

Continuous batching na the whole difference

Imagine say five requests dey hit each server for the same moment, with identical hardware.

Ollama with default settings go run request one reach completion first, then request two, and so on. The fifth caller go wait for four complete generations. Total throughput dey roughly equal to the speed of one generation, because the processor dey work on only one sequence at any time.

vLLM dey decode all five for the same forward pass. To generate one token for five sequences cost barely pass generating one token for one sequence, because the expensive part na reading the model weights from memory, and that read dey shared across the whole batch. Na this same memory-bandwidth fact dey make CPU inference slow: na moving the weights you dey pay for, not the arithmetic.

You fit set OLLAMA_NUM_PARALLEL=4 and get some of this benefit. The cost na memory. Each parallel slot need its own KV cache, and Ollama dey divide the context window across the slots. So four parallel requests against a model configured for 8192 tokens go leave each request with 2048 tokens of context. That 8192 sef na choice, e no be fixed value, so raising num_ctx and sizing the RAM wey e cost na the step wey decide whether four slots fit work at all. vLLM's paged cache na wetin avoid this trade-off, because e dey allocate blocks to a request as the request dey grow. Either way, the limit on how many people one box fit serve at once come down to KV cache size, prefill cost, and queue depth. Na why server wey feel fine for one person dey crawl when five people use am.

Install and serve with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run --verbose llama3.1:8b "Write two sentences about Linux."

The install script dey create one ollama system user, install the binary, and register ollama.service wey bind to 127.0.0.1:11434. The eval rate line wey --verbose print na the real tokens per second for that box. Trust am pass any published figure. One reading from one prompt na starting point, no be capacity number. So timing tokens per second across a concurrency sweep na wetin go show whether the box fit handle the load wey you really expect, and whether renting GPU go better than paying per token.

To increase concurrency, use systemd drop-in so upgrade no go overwrite the change:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl restart ollama
ollama ps

ollama ps dey show wetin load, and e PROCESSOR column dey show the real state. 100% CPU mean say no GPU dey involved. Na this be the honest reason for most reports say Ollama slow. The OLLAMA_KEEP_ALIVE=30m line for that drop-in matter just as much for quiet box, because the default go unload the model after five minutes without request. keeping the model resident between requests na wetin stop the first prompt after one idle hour from paying the full load time again.

Install and serve with vLLM

vLLM need Linux and Python 3.10 to 3.13. Install am for e own virtual environment, because e dey bring one specific PyTorch build:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Then serve one model. The name na Hugging Face repository id, no be short tag:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The first startup slow because e dey download the weights, then e profile the GPU to decide how many KV cache blocks fit. E dey listen on port 8000. Check am before you write any client code:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]}'

If Docker dey already for the machine, the official image go avoid the CUDA dependency work:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

--ipc=host dey required; e no be decoration. PyTorch dey pass tensors between processes through shared memory, and Docker default shared-memory allocation too small for tensor-parallel inference.

The flags wey matter pass for production na --max-model-len (the context window wey you ready pay for), --gpu-memory-utilization (the fraction of the card wey vLLM fit claim, 0.92 by default as of July 2026), --tensor-parallel-size to split one model across plenty GPUs, and --api-key.

Authentication na one flag for vLLM, but Ollama no get am

vLLM go enforce bearer token if you give am:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key token-abc123

The same value fit come from VLLM_API_KEY environment variable. Request wey no get am go receive HTTP 401. But this still no mean say you should publish port 8000 for public interface, because vLLM no get rate limiting and plain HTTP token fit readable while e dey transit. E only mean say server get one concept of caller.

Ollama no get any authentication. No key, no login, no allow-list. Any process wey fit reach 11434 fit run, pull, or delete models. Keep am for loopback and reach am through WireGuard VPN wey you host yourself, or through authenticating reverse proxy wey dey terminate TLS (transport layer security).

Hardware: wetin each one need

Ollama dey run for CPU. 4-bit quantized model dey use roughly half gigabyte RAM for every billion parameters, plus about one gigabyte runtime overhead and more space for context. So, 3B model need around 4 GB free RAM, while 8B model need around 8 GB. Speed for shared vCPU na single digit to low double digit tokens per second. Na memory bandwidth cause this, no be misconfiguration, and no flag fit repair am. To see this calculation for one specific release instead of general estimate, running Nemotron 3.5 Lightning for VPS dey show the exact tag to pull, the RAM wey e hold after loading, and whether CPU-only fast enough to manage.

vLLM assume say GPU dey available. E default path dey serve unquantized weights with 16-bit precision. This one na roughly 2 GB for every billion parameters. So, 8B model need about 16 GB video memory for weights alone, before KV cache wey give you the concurrency wey make you install vLLM. For 24 GB card, cache still get enough usable space. For 16 GB card, e no get enough space. So you either choose smaller model or pass --quantization with quantized checkpoint. CPU backend dey exist, but standard wheels no dey build for am. E also remove the main reason to run vLLM.

So, hardware question dey answer software question most times. If GPU no dey, use Ollama. If you rent GPU but e dey use only 5 percent because requests dey run one after another, use vLLM.

Which one fit your workload

• One person, one CPU VPS, dey draft and summarise: Ollama. The speed dey acceptable and nothing else simple pass am.
• Coding assistant, or MCP server wey dey connect your tools to local model, wey na only you dey call: Ollama. Concurrency of one na the real workload.
• If you dey compare five models this week: Ollama. Pulling and deleting tagged models na exactly wetin e good for, while vLLM need process restart for each model.
• Internal app, chat product, or retrieval pipeline wey get real users: vLLM. Na here batching justify GPU bill.
• Batch job wey dey score hundred thousand documents overnight: vLLM, with high --max-num-seqs. Throughput na the only metric wey matter, and per-document latency no matter.
• Agent platform wey several self-hosted AI agents dey use to reach the model at the same time: vLLM, because agent traffic dey bursty and e naturally dey happen in parallel.

Failure modes, and the strings wey you go see

vLLM no gree start because of KV cache error. The message go show both numbers:

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (8192). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

The model declare context window wey pass the memory wey remain after e load the weights. Reduce am with --max-model-len 8192, or increase --gpu-memory-utilization if nothing else dey use the card. If you push utilisation pass about 0.95, this startup error fit change to CUDA out-of-memory crash later when workload dey run. That one worse.

Ollama print Killed during generation. Linux out-of-memory killer stop the process because the model need more RAM than the box get. Confirm am with sudo dmesg | grep -i oom. The fix na smaller model or model wey get heavier quantization, no be setting.

Ollama dey answer well alone but e stall under load. No error go show anywhere. Requests simply dey take longer as callers increase, because OLLAMA_NUM_PARALLEL=1 dey serialize dem. Long answers make the queue worse. One caller fit hold the only slot until the model decide to stop, so everybody behind am go wait. capping the reply with num_predict fit set upper limit for how long one turn fit hold the server. Increase the parallel setting and accept the smaller per-request context, or move the workload go vLLM.

vLLM return 401 for every call. You start am with --api-key, but the client no send Authorization header. Most OpenAI client libraries send anything wey you pass as the key, so set am there instead of removing the flag.

vLLM say the model no dey found. Ollama dey pull on demand, but vLLM no dey do that. The model field for the request body must match the repository id wey you launch with, or the value of --served-model-name if you set one. Confirm the exact string with curl http://localhost:8000/v1/models.

Run both na reasonable answer

Dem no dey exclude each other. One common setup na to run vLLM for GPU instance to serve the application, while Ollama dey run for the ordinary VPS beside am for local scripts, cron jobs, and to try new model releases. Both endpoints dey OpenAI-compatible, so one client library and base-URL switch fit handle both. Cost control matter pass either engine for this case, because idle GPU still dey bill like busy one, and keeping agent and inference costs predictable na separate discipline from choosing server.

FAQ

vLLM dey faster pass Ollama?

For one request on the same GPU, difference no too big because both dey do the same arithmetic. For plenty concurrent requests, vLLM dey far ahead because continuous batching dey decode every active sequence inside one forward pass, while Ollama default dey run dem one after another. For machine wey na CPU-only, this question no apply: Ollama fit run there, but vLLM practically no fit.

vLLM fit run without GPU?

No, e no go useful. The standard wheels target NVIDIA or AMD GPUs, and the reason vLLM dey exist — to keep accelerator busy with batched requests — no dey apply for CPU. CPU backend dey for development work. For real CPU inference, use Ollama or llama.cpp directly.

Wetin be the difference between Ollama and llama.cpp?

llama.cpp na the inference library, while GGUF na im quantized weight format. Ollama runner build on top of am and add the parts wey llama.cpp leave for you: model registry, automatic download, resident server, systemd unit, and OpenAI-compatible endpoint. Ollama don add im own engine for some newer model families, so the two no be exactly the same underneath again.

How much GPU memory vLLM need for 8B model?

For 16-bit precision, the weights alone dey about 16 GB, roughly 2 GB for every billion parameters, and KV cache need extra space on top. 24 GB card go comfortable. 16 GB card need quantized checkpoint or smaller model. vLLM claim fraction of the card wey --gpu-memory-utilization set, and as of July 2026, the default na 0.92.

I need change my application code to switch between dem?

Usually na only base URL, API key, and model name you need change. Ollama dey serve im OpenAI-compatible surface for http://127.0.0.1:11434/v1 and e ignore the key, while vLLM dey serve http://localhost:8000/v1 and enforce the key if you set one. The model names get different format: llama3.1:8b for Ollama, and full repository id like Qwen/Qwen2.5-1.5B-Instruct for vLLM.