Ollama vs llama.cpp: Which One Fit Your VPS?

Ollama vs llama.cpp: which layer you want run?

Ollama and llama.cpp no be competitors the way the question take imply. llama.cpp na the inference engine: e dey load model file and turn prompt to tokens. Ollama na model manager, background daemon and HTTP API wey dey sit on top of that engine. Ollama README still list llama.cpp as its inference backend (checked 2 August 2026). So the real question na which layer you want operate for your VPS, no be which one dey faster.

Run Ollama when you want service wey go fetch models by name and keep working without attention. Run llama.cpp directly when the box small and you need choose the exact model file, exact context size and exact thread count, because for small VPS, each of those settings dey use memory wey you no get.

Wetin each project really be

llama.cpp na C and C++ implementation for transformer inference wey dey use ggml library. E dey read GGUF files. GGUF (GGML universal file format) na single-file container wey hold the weights, tokeniser, and metadata wey the engine need to run the model. The project dey release separate binaries for separate jobs. llama-server na HTTP server, llama-cli na interactive prompt, and llama-bench dey measure throughput. Releases dey use build number as tag, instead of semantic version. The current tag na b10224, wey dem publish on 2 August 2026, and new tag dey land for most working days.

Ollama na Go program. Background daemon, wey ollama serve start, dey load models and answer HTTP requests, while command line client dey talk to that daemon. Registry for ollama.com dey behind both of dem, and e dey hold prepacked models. Ollama dey use semantic versions, and v0.32.5 ship on 27 July 2026. ollama pull dey fetch GGUF together with prompt template and default parameters, then store am under /usr/share/ollama/.ollama/models for Linux.

Na this packaging be the main difference. Ollama dey choose the quantisation, template, and context length for you, and give you one name to remember. llama.cpp no dey choose anything; e just give you flags.

Axis 1: model and quantisation control

Quantisation dey reduce each weight from 16 or 32 bits go 4, 5, or 8 bits. Na this one make 8 billion parameter model fit inside RAM of ordinary VPS. GGUF naming dey easy to read once you understand the pattern: Q4_K_M mean 4-bit K-quant with medium size. Higher number dey keep more precision, but e dey use more memory.

[
  {
    "label": "Q2_K",
    "file_size_gib": 2.96
  },
  {
    "label": "Q3_K_M",
    "file_size_gib": 3.74
  },
  {
    "label": "Q4_K_M",
    "file_size_gib": 4.58
  },
  {
    "label": "Q5_K_M",
    "file_size_gib": 5.34
  },
  {
    "label": "Q6_K",
    "file_size_gib": 6.14
  },
  {
    "label": "Q8_0",
    "file_size_gib": 7.95
  }
]

Dem na the published file sizes for bartowski/Meta-Llama-3.1-8B-Instruct-GGUF repository on Hugging Face, wey dem read on 2 August 2026 and convert from bytes to GiB. Na 6 builds of one model, and the smallest one na 2.96 GiB compared with 7.95 GiB for the biggest one. The common default, Q4_K_M, na 4.58 GiB. For 4 GiB VPS, na this one choice go decide whether the model go load at all.

With llama.cpp, you specify the file, so na you go pick that row yourself.

llama-server -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 4096 -t 4 --host 127.0.0.1 --port 8080

-c na the context size in tokens, -t na the thread count, and -ngl dey set how many layers go move to GPU (0 for CPU-only box). Nothing dey guessed for you.

With Ollama, the quantisation dey follow the tag wey you pull, and ollama ls dey show wetin actually dey for disk. If registry no carry the build wey you want, import GGUF yourself. Write a Modelfile:

FROM ./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
PARAMETER num_ctx 4096

Then build am and check the result:

ollama create llama31-q4 -f ./Modelfile
ollama ls

Context length na the setting wey dey catch people. Ollama dey pick default from available VRAM, and box wey no get GPU go enter the smallest bucket: 4096 tokens. If you send am 20,000-token document, the extra tokens go drop before model ever see dem, so the answer go confidently wrong about file wey e only read halfway. Raise am with OLLAMA_CONTEXT_LENGTH for daemon, or with PARAMETER num_ctx inside Modelfile. llama.cpp no get any default wey you fit trust too. Set -c explicitly and know wetin you set.

The memory arithmetic wey nobody dey show you

The model file no be the full cost. The KV cache (key/value cache) dey hold one entry for each layer and each context token, and e dey grow as conversation dey longer.

Calculate am for Llama 3.1 8B. The model get 32 layers, 8 key/value heads, and head dimension na 128. Each token stores one key and one value, with 2 bytes for each one inside f16. So, 2 x 8 x 128 x 2 = 4096 bytes for each layer. Across 32 layers, na 128 KiB for each token. Context of 4096 tokens therefore costs 512 MiB, while context of 32,768 tokens costs 4 GiB.

So, Q4_K_M 8B model for 4k context need roughly 4.58 GiB for weights, plus about 0.5 GiB for cache, plus the runtime itself. E no go fit inside 4 GiB RAM. E go fit for 8 GiB with space to work. If you raise the context to 32k for that same 8 GiB machine, the cache alone go use all the available headroom. Monitor am live with free -h while the model dey loaded, and no trust estimate wey you never measure.

Ollama dey multiply this. OLLAMA_NUM_PARALLEL default na 1, and the memory wey model need dey scale with that number multiplied by the context length. If you raise both at the same time, the daemon quietly go request several times the RAM wey you expect.

Axis 2: daemon wey you need operate

The Ollama install script dey write a systemd unit, create one ollama system user, and enable the service. You go get lifecycle management without writing any of am yourself. Configuration dey go through systemd:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -e -u ollama

OLLAMA_KEEP_ALIVE dey matter pass for CPU VPS than for anywhere else. By default, models dey stay for memory for 5 minutes, then dem go unload. The next request go need read the whole file from disk again before e fit answer. So, for slow storage, 4.58 GiB reload fit turn two-second reply to thirty-second one. Long keep-alive go fix the latency, but e go permanently use the RAM. Both options get real cost. Choose the one wey go hurt less.

llama.cpp no provide daemon, so you go write the unit yourself as /etc/systemd/system/llama-server.service:

[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
ExecStart=/usr/local/bin/llama-server -m /srv/models/model-Q4_K_M.gguf -c 4096 -t 4 --host 127.0.0.1 --port 8080
Restart=always
RestartSec=3
User=llama

[Install]
WantedBy=multi-user.target

Enable am with sudo systemctl enable --now llama-server. The process go then hold the model for the whole time e dey run. Nothing go unload am when e idle. This mean say no reload surprise, but you no fit reclaim the memory unless you stop the service. If writing units still new to you, na the same pattern as running your own services under systemd for VPS.

Axis 3: the API wey your app go talk to

This axis don narrow well-well. Both projects now dey speak OpenAI chat format, so most client libraries fit work with either one after you change only the base URL.

Ollama dey listen on 127.0.0.1:11434. Its OpenAI-compatible route na http://localhost:11434/v1/chat/completions, and e still get native API for /api/chat alongside am. Dem also document Anthropic-compatible route.

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama31-q4", "messages": [{"role": "user", "content": "Say this is a test"}]}'

llama-server dey listen on 127.0.0.1:8080 and e serve /v1/chat/completions, /v1/completions and /v1/embeddings, plus its own /completion endpoint and built-in web UI. E also expose operational routes wey Ollama no get: /health for readiness probe, /props for the settings of the loaded model, /slots for wetin each request slot dey do, and /metrics for Prometheus format. If you plan monitor this service, na this difference fit decide am.

Neither server go enable authentication for you. Both dey use loopback by default for good reason. Reach dem through SSH tunnel or from behind reverse proxy, and never open 11434 or 8080 to internet.

Wetin CPU-only VPS fit honestly do

CPU-only VPS dey run small models slowly. Na the honest summary be that, and the useful part na to know where the limit dey. Measure before you design anything around am:

llama-bench -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128

The pp column na prompt processing speed, and the tg column na token generation speed, both for tokens per second. For shared vCPU plan, 8B model for Q4_K_M usually dey reach low single digits for tg. Prompt processing na the part wey dey cause wahala: the whole prompt dey process before the first output token show, so long system prompt dey add wait to every request.

CPU fit handle: 1B to 4B model wey dey do classification, extraction, short summaries, or routing. Replies dey arrive within seconds, and the memory fit normal plan. CPU no fit handle: interactive chat for reading speed, coding assistants, long-document work, or anything wey agent loop dey make many calls one after another. Loop wey make twelve calls for four seconds each go take one minute before e produce anything.

Two options dey when the numbers no work. If na concurrency be the problem, meaning many users dey hit one model at the same time, engine choice go change, and the comparison of Ollama and vLLM for concurrent serving cover that matter. If na raw speed be the problem, the answer na VPS wey get GPU attached, where -ngl go start to mean something. Before either one, get baseline for the hardware itself, because disk and memory bandwidth affect load time almost as much as CPU. Repeatable VPS benchmark worth the one hour.

Install llama.cpp wey lock to one build

Both projects dey change every week, so record the version wey you deploy. The upstream one-liner dey install the current build:

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

To lock one specific build, take the prebuilt tarball from the releases page instead. Build b10224 na the current tag as of 2 August 2026:

curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10224/llama-b10224-bin-ubuntu-x64.tar.gz
tar xf llama-b10224-bin-ubuntu-x64.tar.gz
find . -type f -name 'llama-server'

Or build that same tag from source:

sudo apt update && sudo apt install -y build-essential cmake git libssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10224
cmake -B build
cmake --build build --config Release -j $(nproc)

libssl-dev na the documented dependency for the HTTPS features. The compile fit take several minutes and e need more RAM pass wetin the smallest plans get, so build am for a bigger box and copy the binaries if the small one no get enough resources.

Install Ollama, pin am to one version

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.32.5 sh
ollama -v

The script dey read OLLAMA_VERSION, so you fit hold one release wey you don confirm say e good, instead of make e collect anything wey release this morning. v0.32.5 come out on 27 July 2026. Manual option dey too if you no wan pipe script enter shell:

sudo rm -rf /usr/lib/ollama
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr
ollama -v

Manual method no create systemd unit or service user, so na you go add dem yourself. The complete Ollama for VPS walkthrough explain how to set up the service step by step.

Failure modes, and the strings wey you go see

Ollama no gree load the model. ollama run dey return line wey get this shape:

Error: model requires more system memory (5.6 GiB) than is available (3.2 GiB)

Ollama dey check the size before e load am, so e fail quick and talk why. Move go one quantisation row down, reduce the context length, or choose smaller model.

llama.cpp no fail, e dey crawl. llama.cpp dey memory-map the GGUF by default, so file wey pass RAM still fit start. Kernel go dey page weights in and out from disk for every token, and generation go drop reach seconds per token while disk dey pinned for 100 percent. Pass --no-mmap to force real allocation, so e fail immediately instead of degrading. When kernel step in, dmesg dey show the reason:

Out of memory: Killed process 1234 (llama-server)

The model file no go load at all. GGUF wey dem build for model family newer than your engine go return error wey name the architecture wey e no know:

error loading model architecture: unknown model architecture: 'qwen3next'

The fix na engine upgrade, no be different file. Na the price of pinning, and na why you write down the build number. You need know wetin you dey upgrade from.

The API dey answer locally but e no answer from your app. Ollama dey bind 127.0.0.1:11434, so another host go get connection refused. Set OLLAMA_HOST=0.0.0.0:11434 through systemctl edit ollama only when the port dey behind firewall or private network, because the API no get authentication in front of am.

The first reply after pause dey very slow. The 5 minute idle unload don happen, and the model dey read from disk again. ollama ps wey you run just before the request no dey show anything loaded, and this confirm am. Increase OLLAMA_KEEP_ALIVE.

So, which one you suppose run?

Run Ollama when you want make e manage models for you and provide an OpenAI-shaped endpoint without extra work. Na the better default for first deployment, and for anything wey model choice go dey change often.

Run llama.cpp directly when memory tight reach where you need choose the quantisation row by yourself, when you want /health, /slots and /metrics for monitoring, or when you need a flag wey Ollama no expose. Na the honest choice for VPS where the model barely fit, because the settings wey make am fit na exactly the ones wey Ollama choose for you.

Running both dey normal. Use Ollama for experiments, and llama.cpp for the one model wey you put for production and no ever want see change.

FAQ

Ollama na just wrapper around llama.cpp?

E near, but the wrapper dey do real work. Ollama README list llama.cpp as its inference backend (checked 2 August 2026). Ollama add model registry, prompt template wey dey turn chat messages into prompt, default sampling parameters, daemon wey dey unload model when e idle, and HTTP API. When you compare tokens per second with the same settings, na the same engine you dey compare with itself. Wetin you really dey choose between na the management layer.

Which one fast pass for CPU-only VPS?

Dem share the same engine, so if model file, quantisation, context size, and thread count dey the same, dem go get almost the same result. Differences wey people report normally come from different defaults, especially context length and thread count, no be from the engine. Measure am with llama-bench -m <file> -p 512 -n 128 and compare the tg column for your own box before you trust any published figure.

I fit use my own GGUF file with Ollama?

Yes. Put the file for the server, write a Modelfile wey the first line na FROM ./your-model.gguf, add any PARAMETER lines wey you need, like num_ctx, then run ollama create your-name -f ./Modelfile. ollama ls go list am together with anything wey you pull from the registry. Na so you fit use quantisation wey the registry no carry.

How much RAM I need for 8B model?

Plan for the file size, plus KV cache, plus the runtime. Q4_K_M build of Llama 3.1 8B na about 4.58 GiB for disk, and 4096 token context dey add roughly 512 MiB cache, so 8 GiB RAM dey comfortable, but 4 GiB no reach. The cache dey scale with context: the same model for 32,768 token context need about 4 GiB cache by itself. With Ollama, remember say the requirement dey also scale with OLLAMA_NUM_PARALLEL.