Ollama or vLLM: which one fit run your LLM?

Ollama vs vLLM, for one paragraph

Ollama na model manager wey server join am: e dey download quantized weights, load dem, and answer for 127.0.0.1:11434, for CPU if na wetin the box get. vLLM na throughput engine: e dey keep GPU busy with plenty requests wey dey run at the same time, and e no be the correct tool for machine wey no get GPU. Na so the decision be. One person wey dey talk to local assistant na Ollama job. Application wey dey serve team na vLLM job.

Both dey use OpenAI-compatible HTTP API, so client code fit move between dem by changing base URL. Na not API be the difference. The difference na wetin dey happen when second request enter while the first one still dey generate tokens.

Wetin Ollama really be

Ollama na convenience layer. E give you model registry (ollama pull llama3.1:8b), local store for weights, chat prompt, systemd service, and HTTP API, all from one install command. The models wey e serve na GGUF files, usually dem dey quantize am to 4-bit. Na why 7B or 8B model fit dey around 5 GB for disk instead of 16 GB. Quantization na wetin make CPU inference possible at all.

Its runner dey built on llama.cpp, the C++ inference library wey make GGUF quantization practical for ordinary hardware. Ollama don add its own engine for some newer model families, but llama.cpp still be the substrate under most of wetin e serve. So when people compare Ollama with llama.cpp, na mostly ergonomics layer dem dey compare with the thing wey e wrap.

The design target na one user. As of July 2026, default for OLLAMA_NUM_PARALLEL na 1. This mean say one model dey process one request at a time, while everything else dey wait for queue wey hold 512 entries by default (OLLAMA_MAX_QUEUE). You fit increase the parallel setting, and the section below explain the cost. If you never run Ollama before, start with how to host Ollama for VPS and keep port 11434 closed, because the API no get any authentication at all.

Wetin vLLM really be

vLLM na inference server and nothing else. E no manage model library, e no get chat prompt, and e no go pull model for you when request time reach. You name one Hugging Face repository when you launch am, e load that one model, and e serve am until you stop the process.

Wetin you gain from this narrow purpose na throughput. Two mechanisms dey make am work. PagedAttention dey store the KV cache (key-value cache, the per-token attention state wey model dey keep for every active request) inside fixed-size blocks, like how operating system dey page memory. Request no longer need one big contiguous reservation wey size for the worst case, so memory wey previously dey reserved and unused fit become available for more concurrent requests. Continuous batching dey allow new request join the batch wey dey run for the next decoding step instead of waiting for the current batch to finish. Sequence wey finish dey leave the batch immediately, and dem refill the slot.

The practical result be say: for one GPU, when concurrent users increase from one to thirty, total tokens per second dey rise sharply, while speed for each user dey reduce much less than you go expect. Under Ollama default, when users increase from one to thirty, e just make twenty-nine people wait.

Continuous batching na the main difference

Imagine say five requests reach each server for the same time on identical hardware.

With default settings, Ollama runs request one finish first, then request two, and so on. The fifth caller go wait for four complete generations. Total throughput dey roughly equal to the speed of one generation, because the processor dey work on only one sequence at any time.

vLLM dey decode all five for the same forward pass. To generate one token for five sequences dey cost almost the same as generating one token for one sequence, because the expensive part na to read the model weights from memory, and the whole batch dey share that read. Na the same memory-bandwidth fact dey make CPU inference slow: you dey pay to move weights, not for the arithmetic.

You fit set OLLAMA_NUM_PARALLEL=4 and get some of this benefit. The cost na memory. Each parallel slot need its own KV cache, and Ollama dey divide the context window across the slots. So four parallel requests against a model configured for 8192 tokens go leave 2048 tokens of context for each request. vLLM's paged cache na wetin dey avoid this trade-off, because blocks dey allocate to a request as the request dey grow.

Install and serve with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run --verbose llama3.1:8b "Write two sentences about Linux."

The install script dey create one ollama system user, install the binary, and register ollama.service wey bind to 127.0.0.1:11434. The eval rate line wey --verbose print na your real tokens per second for that box. Trust am pass any published figure.

To increase concurrency, use one systemd drop-in so upgrade no go overwrite the change:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl restart ollama
ollama ps

ollama ps dey show wetin load, and the PROCESSOR column dey tell the truth. 100% CPU mean say no GPU dey involved. Na the honest reason for most reports say Ollama slow.

Install and serve with vLLM

vLLM need Linux and Python 3.10 to 3.13. Install am inside e own virtual environment, because e dey bring specific PyTorch build:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Then serve one model. The name na Hugging Face repository id, e no be short tag:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Startup slow the first time, because e download the weights first, then e profile the GPU to decide how many KV cache blocks fit. E dey listen on port 8000. Check am before you write any client code:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]}'

If Docker dey on the box already, the official image go remove the CUDA dependency work:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

--ipc=host dey required, e no be decoration: PyTorch dey pass tensors between processes through shared memory, and Docker default shared-memory allocation too small for tensor-parallel inference.

The flags wey matter pass for production na --max-model-len (the context window wey you ready pay for), --gpu-memory-utilization (the fraction of the card wey vLLM fit claim, 0.92 by default as of July 2026), --tensor-parallel-size to split one model across several GPUs, and --api-key.

Authentication na one flag for vLLM and e no dey for Ollama

vLLM dey enforce bearer token if you give am:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key token-abc123

The same value fit come from the VLLM_API_KEY environment variable. Request wey no get am go receive HTTP 401. But this still no be reason to publish port 8000 for public interface, because vLLM no get rate limiting and plain HTTP token fit dey readable as e dey pass through network. But e mean say server get concept of who dey call am.

Ollama no get any. No key, no login, no allow-list. Any process wey fit reach 11434 fit run, pull, or delete models. Keep am for loopback and reach am through WireGuard VPN wey you host yourself, or through authenticating reverse proxy wey dey terminate TLS (transport layer security).

Hardware: wetin each one need

Ollama dey run for CPU. A 4-bit quantized model dey use roughly half gigabyte of RAM for every billion parameters, plus about one gigabyte runtime overhead and more space for context. So, 3B model need around 4 GB free, while 8B model need around 8 GB. Speed for shared vCPU na single-digit to low double-digit tokens per second. Na memory bandwidth cause am, no be misconfiguration, and no flag fit repair am.

vLLM dey assume say GPU dey available. E default path dey serve unquantized weights for 16-bit precision. This na roughly 2 GB for every billion parameters. So, 8B model need about 16 GB video memory for weights alone, before KV cache wey dey provide the concurrency wey make you install vLLM. For 24 GB card, cache still get workable space. For 16 GB card, e no get enough space. So you either choose smaller model or pass --quantization with quantized checkpoint. CPU backend dey exist, but standard wheels no build for am, and e remove the main reason to run vLLM.

So, hardware question dey answer software question most times. If GPU no dey, use Ollama. If rented GPU dey use only 5 percent because requests dey run one after another, use vLLM.

Which one fit your workload

• One person, a CPU VPS, dey draft and summarise: Ollama. The speed dey okay, and nothing else easy pass am.
• A coding assistant, or an MCP server wey dey connect your tools to a local model, wey na only you dey call: Ollama. Concurrency of one na the real workload.
• You dey compare five models this week: Ollama. Pulling and deleting tagged models na exactly wetin e good for, while vLLM need process restart for each model.
• An internal app, chat product, or retrieval pipeline wey get real users: vLLM. Na here batching make GPU bill worth am.
• A batch job wey dey score one hundred thousand documents overnight: vLLM, with high --max-num-seqs. Throughput na the only metric wey matter, and per-document latency no matter.
• An agent platform wey several self-hosted AI agents dey use to reach the model at the same time: vLLM, because agent traffic dey come in bursts and e dey run parallel by nature.

Failure modes, with the strings you will see

vLLM no gree start and e show KV cache error. The message go name both numbers:

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (8192). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

The model declare context window wey pass the memory wey remain after e load the weights. Reduce am with --max-model-len 8192, or increase --gpu-memory-utilization if nothing else dey use the card. If you push utilisation pass about 0.95, e fit change this startup error to CUDA out-of-memory crash later when load dey high. That one worse.

Ollama print Killed for inside generation. Linux out-of-memory killer stop the process because the model need more RAM than the box get. Confirm am with sudo dmesg | grep -i oom. The fix na smaller model or model wey dem quantize more, no be setting.

Ollama dey answer well alone but e stall when load dey high. No error dey show anywhere. Requests just dey take longer as callers increase, because OLLAMA_NUM_PARALLEL=1 dey serialize dem. Increase am and accept the smaller context for each request, or move the workload go vLLM.

vLLM dey return 401 for every call. You start am with --api-key and the client no dey send Authorization header. Most OpenAI client libraries send anything wey you pass as the key, so set am there instead of removing the flag.

vLLM talk say e no find the model. Ollama dey pull on demand, but vLLM no dey do am. The model field for the request body must match the repository id wey you launch with, or the value of --served-model-name if you set one. Confirm the exact string with curl http://localhost:8000/v1/models.

Running both na reasonable answer

Dem no dey exclude each other. One common setup na vLLM for GPU instance wey dey serve the application, while Ollama dey run for the ordinary VPS beside am for local scripts, cron jobs, and testing new model releases. Both endpoints dey OpenAI-compatible, so one client library plus base-URL switch fit handle both. Cost control dey matter pass either engine for here, because idle GPU still dey bill the same way as busy one, and to keep agent and inference costs predictable na separate work from choosing server.

FAQ

vLLM dey faster pass Ollama?

For one request for the same GPU, the difference no too big, because both dey do the same arithmetic. For many requests wey dey happen together, vLLM dey far ahead, because continuous batching dey decode every active sequence for one forward pass, while Ollama default dey run dem one after another. For machine wey na CPU-only, this question no apply: Ollama dey run there, but vLLM practically no dey run.

vLLM fit run without GPU?

E no go work well. The standard wheels target NVIDIA or AMD GPUs, and the reason vLLM exist, wey be to keep accelerator full with batched requests, no dey apply for CPU. CPU backend dey for development work. For real CPU inference, use Ollama or llama.cpp directly.

Wetin be the difference between Ollama and llama.cpp?

llama.cpp na the inference library, and GGUF na the quantized weight format wey e use. Ollama runner build on top of am and add the parts wey llama.cpp leave for you: model registry, automatic download, resident server, systemd unit, and OpenAI-compatible endpoint. Ollama don add its own engine for some newer model families, so the two no be exactly the same underneath again.

How much GPU memory vLLM need for 8B model?

For 16-bit precision, the weights alone dey need about 16 GB, roughly 2 GB for every billion parameters, and KV cache need extra space on top. 24 GB card go work comfortably. 16 GB card need quantized checkpoint or smaller model. vLLM claim the fraction of the card wey --gpu-memory-utilization set, and as of July 2026, the default na 0.92.

I need change my application code to switch between dem?

Usually, na only the base URL, API key, and model name you need change. Ollama dey serve its OpenAI-compatible surface for http://127.0.0.1:11434/v1 and e ignore the key, while vLLM dey serve http://localhost:8000/v1 and e enforce the key if you set one. The model names get different format: llama3.1:8b for Ollama, and full repository id like Qwen/Qwen2.5-1.5B-Instruct for vLLM.