Ollama vs llama.cpp on a VPS

Ollama vs llama.cpp: which layer do you want to run?

Ollama and llama.cpp are not competitors in the way the question implies. llama.cpp is the inference engine: it loads a model file and turns a prompt into tokens. Ollama is a model manager, a background daemon and an HTTP API sitting on top of that engine. Ollama's README still lists llama.cpp as its inference backend (checked 2 August 2026). So the real question is which layer you want to operate on your VPS, not which one is faster.

Run Ollama when you want a service that fetches models by name and keeps working without attention. Run llama.cpp directly when the box is small and you need to pick the exact model file, the exact context size and the exact thread count, because on a small VPS every one of those settings costs memory you do not have.

What each project actually is

llama.cpp is a C and C++ implementation of transformer inference built on the ggml library. It reads GGUF files. GGUF (GGML universal file format) is a single-file container holding the weights, the tokeniser and the metadata the engine needs to run the model. The project ships separate binaries for separate jobs. llama-server is an HTTP server, llama-cli is an interactive prompt, and llama-bench measures throughput. Releases are tagged by build number rather than by semantic version. The current tag is b10224, published on 2 August 2026, and a new tag lands most working days.

Ollama is a Go program. A background daemon, started with ollama serve, loads models and answers HTTP requests, and a command line client talks to that daemon. Behind both sits a registry at ollama.com holding prepacked models. Ollama uses semantic versions, and v0.32.5 shipped on 27 July 2026. ollama pull fetches a GGUF along with a prompt template and a set of default parameters, then stores it under /usr/share/ollama/.ollama/models on Linux.

That packaging is the whole difference. Ollama decides the quantisation, the template and the context length for you, and gives you one name to remember. llama.cpp decides nothing and gives you flags.

Axis 1: model and quantisation control

Quantisation shrinks each weight from 16 or 32 bits down to 4, 5 or 8 bits. That is what makes an 8 billion parameter model fit in the RAM of an ordinary VPS. The GGUF naming is readable once you know the pattern: Q4_K_M means 4-bit K-quant, medium size. A higher number keeps more precision and costs more memory.

[
  {
    "label": "Q2_K",
    "file_size_gib": 2.96
  },
  {
    "label": "Q3_K_M",
    "file_size_gib": 3.74
  },
  {
    "label": "Q4_K_M",
    "file_size_gib": 4.58
  },
  {
    "label": "Q5_K_M",
    "file_size_gib": 5.34
  },
  {
    "label": "Q6_K",
    "file_size_gib": 6.14
  },
  {
    "label": "Q8_0",
    "file_size_gib": 7.95
  }
]

Those are the published file sizes in the bartowski/Meta-Llama-3.1-8B-Instruct-GGUF repository on Hugging Face, read on 2 August 2026 and converted from bytes to GiB. 6 builds of one model, and the smallest is 2.96 GiB against 7.95 GiB for the largest. The common default, Q4_K_M, is 4.58 GiB. On a 4 GiB VPS that single choice decides whether the model loads at all.

With llama.cpp you name the file, so you pick that row yourself.

llama-server -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 4096 -t 4 --host 127.0.0.1 --port 8080

-c is the context size in tokens, -t is the thread count, and -ngl sets how many layers move to a GPU (0 on a CPU-only box). Nothing is guessed for you.

With Ollama the quantisation rides along with the tag you pull, and ollama ls shows what you actually have on disk. When the registry does not carry the build you want, import a GGUF yourself. Write a Modelfile:

FROM ./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
PARAMETER num_ctx 4096

Then build it and check the result:

ollama create llama31-q4 -f ./Modelfile
ollama ls

Context length is the setting that catches people. Ollama picks its default from available VRAM, and a box with no GPU falls into the smallest bucket: 4096 tokens. Send it a 20,000 token document and the extra tokens are dropped before the model ever sees them, so the answer is confidently wrong about a file it only half read. Raise it with OLLAMA_CONTEXT_LENGTH on the daemon, or with PARAMETER num_ctx in a Modelfile. llama.cpp has no default worth trusting either. Set -c explicitly and know what you set.

The memory arithmetic nobody shows you

The model file is not the whole cost. The KV cache (key/value cache) holds one entry per layer per token of context, and it grows as the conversation grows.

Work it out for Llama 3.1 8B. The model has 32 layers, 8 key/value heads and a head dimension of 128. Each token stores both a key and a value at 2 bytes each in f16, so 2 x 8 x 128 x 2 = 4096 bytes per layer. Across 32 layers that is 128 KiB per token. A 4096 token context therefore costs 512 MiB, and a 32,768 token context costs 4 GiB.

So a Q4_K_M 8B model at 4k context needs roughly 4.58 GiB for weights, plus about 0.5 GiB of cache, plus the runtime itself. It does not fit in 4 GiB of RAM. It fits in 8 GiB with room to work. Raise the context to 32k on that same 8 GiB box and the cache alone eats the headroom. Watch it live with free -h while the model is loaded, and do not trust an estimate you did not measure.

Ollama multiplies this. OLLAMA_NUM_PARALLEL defaults to 1, and the memory a model needs scales with that number times the context length. Raise both at once and the daemon quietly asks for several times the RAM you expected.

Axis 2: the daemon you have to operate

The Ollama install script writes a systemd unit, creates an ollama system user and enables the service. You get lifecycle management without writing any of it. Configuration goes through systemd:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -e -u ollama

OLLAMA_KEEP_ALIVE matters more on a CPU VPS than anywhere else. Models are kept in memory for 5 minutes by default and then unloaded. The next request has to read the whole file back from disk before it can answer, so a 4.58 GiB reload turns a two second reply into a thirty second one on slow storage. A long keep-alive fixes the latency and permanently spends the RAM. Both are real costs. Pick the one that hurts less.

llama.cpp gives you no daemon, so you write the unit yourself as /etc/systemd/system/llama-server.service:

[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
ExecStart=/usr/local/bin/llama-server -m /srv/models/model-Q4_K_M.gguf -c 4096 -t 4 --host 127.0.0.1 --port 8080
Restart=always
RestartSec=3
User=llama

[Install]
WantedBy=multi-user.target

Enable it with sudo systemctl enable --now llama-server. The process then holds the model for its whole lifetime. Nothing unloads on idle, which means no reload surprise and no way to reclaim the memory short of stopping the service. If writing units is new ground, it is the same pattern as running your own services under systemd on a VPS.

Axis 3: the API your app will talk to

This axis has narrowed a lot. Both projects now speak the OpenAI chat format, so most client libraries work against either one after only a base URL change.

Ollama listens on 127.0.0.1:11434. Its OpenAI-compatible route is http://localhost:11434/v1/chat/completions, and it keeps a native API at /api/chat alongside. An Anthropic-compatible route is documented as well.

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama31-q4", "messages": [{"role": "user", "content": "Say this is a test"}]}'

llama-server listens on 127.0.0.1:8080 and serves /v1/chat/completions, /v1/completions and /v1/embeddings, plus its own /completion endpoint and a built-in web UI. It also exposes operational routes that Ollama does not: /health for a readiness probe, /props for the loaded model's settings, /slots for what each request slot is doing, and /metrics in Prometheus format. If you plan to monitor this service, that difference is probably what decides it.

Neither server switches on authentication for you. Both default to loopback for good reason. Reach them over an SSH tunnel or from behind a reverse proxy, and never open 11434 or 8080 to the internet.

What a CPU-only VPS can honestly do

A CPU-only VPS runs small models slowly. That is the honest summary, and the useful part is knowing where the line sits. Measure before you design anything around it:

llama-bench -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128

The pp column is prompt processing speed and the tg column is token generation speed, both in tokens per second. On a shared vCPU plan an 8B model at Q4_K_M usually lands in the low single digits for tg. Prompt processing is the part that hurts: the whole prompt is processed before the first output token appears, so a long system prompt adds a wait to every single request.

Usable on CPU: a 1B to 4B model doing classification, extraction, short summaries or routing. Replies arrive in seconds and the memory fits a normal plan. Not usable on CPU: interactive chat at reading speed, coding assistants, long-document work, or anything with an agent loop making many calls in sequence. A loop that makes twelve calls at four seconds each takes a minute before it produces anything.

There are two exits when the numbers do not work. If the problem is concurrency, many users hitting one model at once, the engine choice changes, and the comparison of Ollama against vLLM for concurrent serving covers that ground. If the problem is raw speed, the answer is a VPS with a GPU attached, where -ngl starts to mean something. Before either, get a baseline for the hardware itself, because disk and memory bandwidth shape load time as much as the CPU does. A repeatable VPS benchmark is worth the hour.

Install llama.cpp, pinned to a build

Both projects move weekly, so record the version you deployed. The upstream one-liner installs the current build:

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

To pin a specific build, take the prebuilt tarball from the releases page instead. Build b10224 is the current tag as of 2 August 2026:

curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10224/llama-b10224-bin-ubuntu-x64.tar.gz
tar xf llama-b10224-bin-ubuntu-x64.tar.gz
find . -type f -name 'llama-server'

Or build that same tag from source:

sudo apt update && sudo apt install -y build-essential cmake git libssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10224
cmake -B build
cmake --build build --config Release -j $(nproc)

libssl-dev is the documented dependency for the HTTPS features. The compile takes several minutes and wants more RAM than the smallest plans have, so build on a larger box and copy the binaries if the small one runs out.

Install Ollama, pinned to a version

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.32.5 sh
ollama -v

The script reads OLLAMA_VERSION, so you can hold a known-good release instead of taking whatever shipped this morning. v0.32.5 was published on 27 July 2026. There is also a manual path if you would rather not pipe a script into a shell:

sudo rm -rf /usr/lib/ollama
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr
ollama -v

The manual route does not create the systemd unit or the service user, so you add those yourself. The full Ollama on a VPS walkthrough covers that service setup step by step.

Failure modes, and the strings you will see

Ollama refuses to load the model. ollama run returns a line of this shape:

Error: model requires more system memory (5.6 GiB) than is available (3.2 GiB)

Ollama checks the size before loading, so it fails fast and says why. Move down one quantisation row, lower the context length, or pick a smaller model.

llama.cpp does not fail, it crawls. llama.cpp memory-maps the GGUF by default, so a file larger than RAM still starts. The kernel then pages weights in and out from disk on every token, and generation drops to seconds per token with the disk pinned at 100 percent. Pass --no-mmap to force a real allocation so it fails immediately instead of degrading. When the kernel does step in, dmesg shows the reason:

Out of memory: Killed process 1234 (llama-server)

The model file will not load at all. A GGUF built for a model family newer than your engine gives an error naming the architecture it does not know:

error loading model architecture: unknown model architecture: 'qwen3next'

The fix is an engine upgrade, not a different file. This is the price of pinning, and it is why you write down the build number. You need to know what you are upgrading from.

The API answers locally but not from your app. Ollama binds 127.0.0.1:11434, so another host gets connection refused. Set OLLAMA_HOST=0.0.0.0:11434 through systemctl edit ollama only when the port sits behind a firewall or a private network, because the API has no authentication in front of it.

The first reply after a pause is very slow. The 5 minute idle unload happened and the model is being read from disk again. ollama ps run just before the request shows nothing loaded, which confirms it. Raise OLLAMA_KEEP_ALIVE.

So which one should you run?

Run Ollama when you want models managed for you and an OpenAI-shaped endpoint with no work. It is the right default for a first deployment, and for anything where the model choice will keep changing.

Run llama.cpp directly when memory is tight enough that you need to pick the quantisation row yourself, when you want /health, /slots and /metrics for monitoring, or when you need a flag Ollama does not expose. It is the honest choice on a VPS where the model barely fits, because the settings that make it fit are exactly the ones Ollama picks on your behalf.

Running both is normal. Ollama for experiments, llama.cpp for the one model you put in production and never want to see move.

FAQ

Is Ollama just a wrapper around llama.cpp?

Close, but the wrapper does real work. Ollama's README lists llama.cpp as its inference backend (checked 2 August 2026). On top of it Ollama adds a model registry, the prompt template that turns chat messages into a prompt, a set of default sampling parameters, a daemon with idle unloading, and an HTTP API. When you compare tokens per second at identical settings, you are comparing the same engine to itself. What you actually choose between is the management layer.

Which is faster on a CPU-only VPS?

They share the engine, so at the same model file, quantisation, context size and thread count they land close together. Differences people report usually come from different defaults, most often context length and thread count, rather than from the engine. Measure it with llama-bench -m <file> -p 512 -n 128 and compare the tg column on your own box before believing any published figure.

Can I use my own GGUF file with Ollama?

Yes. Put the file on the server, write a Modelfile whose first line is FROM ./your-model.gguf, add any PARAMETER lines you need such as num_ctx, then run ollama create your-name -f ./Modelfile. ollama ls will list it next to anything you pulled from the registry. This is how you use a quantisation the registry does not carry.

How much RAM do I need for an 8B model?

Budget the file size, plus the KV cache, plus the runtime. A Q4_K_M build of Llama 3.1 8B is about 4.58 GiB on disk, and a 4096 token context adds roughly 512 MiB of cache, so 8 GiB of RAM is comfortable and 4 GiB is not enough. The cache scales with context: the same model at a 32,768 token context needs about 4 GiB of cache on its own. With Ollama, remember the requirement also scales with OLLAMA_NUM_PARALLEL.