Ollama or llama.cpp for CPU VPS: Which One Fit?

Ollama vs llama.cpp: which layer you wan run?

Ollama and llama.cpp no be competitors the way the question imply. llama.cpp na the inference engine: e load model file and turn prompt into tokens. Ollama na model manager, background daemon, and HTTP API wey dey sit on top of that engine. Ollama README still list llama.cpp as e inference backend (checked 27 October 2026). So the real question na which layer you wan operate for your VPS, no be which one fast pass.

Run Ollama when you want service wey go fetch models by name and keep working without attention. Run llama.cpp directly when the box small and you need choose the exact model file, exact context size, and exact thread count, because for small VPS every one of these settings dey use memory wey you no get.

Wetín each project really be

llama.cpp na C and C++ implementation for transformer inference wey dey use ggml library. E dey read GGUF files. GGUF (GGML universal file format) na single-file container wey hold the weights, tokeniser, and metadata wey engine need to run the model. The project dey release separate binaries for separate jobs. llama-server na HTTP server, llama-cli na interactive prompt, and llama-bench dey measure throughput. The project don begin release semantic versions (MAJOR.MINOR.PATCH), and v0.7.0, wey dem publish on 15 October 2026, na the latest one as of 27 October 2026. Build-numbered nightly tags still dey land alongside dem, several times each day, and the newest one on 27 October 2026 na b11913.

Ollama na Go program. A background daemon, wey ollama serve start, dey load models and answer HTTP requests, while command line client dey talk to that daemon. Behind both of dem, registry for ollama.com dey hold prepacked models. Ollama dey use semantic versions. v0.32.5 ship on 27 July 2026, and v0.40.12 release on 27 October 2026 at 00:07 UTC. ollama pull dey fetch a GGUF together with prompt template and default parameters, then store am under /usr/share/ollama/.ollama/models for Linux. Those files dey sit for root disk and each one fit reach several gigabytes, so if VPS get 25 GB root volume, e good make you know wetin pull leave behind and how to move the model directory go another place before the third download fill am.

Na that packaging be the main difference. Ollama dey decide the quantisation, template, and context length for you, and give you one name to remember. llama.cpp dey expose those choices as flags, with automatic defaults for settings wey you leave unset.

Axis 1: model and quantisation control

Quantisation dey store weights for fewer bits than the original model wey get 16 or 32 bits. Common GGUF builds dey average roughly 4, 5, or 8 bits per weight. Precision no dey uniform across the file. For llama.cpp quantiser source (read 27 October 2026), Q4_K_M build dey keep most tensors for 4-bit. But e dey give the output layer, plus attention value and feed-forward down tensors for about half the layers, 6-bit Q6_K instead. Small tensors like norms no dey quantise at all. Na this make 8 billion parameter model fit inside the RAM of ordinary VPS. GGUF naming dey easy to read once you sabi the pattern: Q4_K_M mean 4-bit K-quant with medium size. Higher number dey keep more precision and dey use more memory.

[
  {
    "label": "Q2_K",
    "file_size_gib": 2.96
  },
  {
    "label": "Q3_K_M",
    "file_size_gib": 3.74
  },
  {
    "label": "Q4_K_M",
    "file_size_gib": 4.58
  },
  {
    "label": "Q5_K_M",
    "file_size_gib": 5.34
  },
  {
    "label": "Q6_K",
    "file_size_gib": 6.14
  },
  {
    "label": "Q8_0",
    "file_size_gib": 7.95
  }
]

These na the published file sizes for bartowski/Meta-Llama-3.1-8B-Instruct-GGUF repository on Hugging Face, read on 2 August 2026 and converted from bytes to GiB. 6 builds of one model dey listed, and the smallest one na 2.96 GiB against 7.95 GiB for the largest. The common default, Q4_K_M, na 4.58 GiB. For 4 GiB VPS, this single choice decide whether the model go load at all. Size na only half of the decision, because row wey you fit afford no automatically mean say e worth using. wetin Q4, Q8 and fp16 really cost you for answer quality na wetin go tell you whether the extra gigabytes buy anything wey you go notice.

With llama.cpp, you name the file, so na you dey pick the row.

llama-server -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 4096 -t 4 --host 127.0.0.1 --port 8080

-c na the context size for tokens, -t na the thread count, and -ngl set how many layers go move to GPU (0 for CPU-only box). Settings wey you no set get defaults, and --fit dey on by default to adjust arguments wey you no set so dem fit device memory.

For Ollama, the quantisation dey come with the tag wey you pull, and ollama ls show wetin you actually get for disk. If registry no carry the build wey you want, import GGUF by yourself. Write Modelfile:

FROM ./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
PARAMETER num_ctx 4096

Then build am and check the result:

ollama create llama31-q4 -f ./Modelfile
ollama ls

Context length na the setting wey dey catch people. Ollama dey choose default based on available VRAM, and box wey no get GPU dey enter the smallest bucket: 4096 tokens. If you send 20,000 token document, the extra tokens go drop before model even see dem. So the answer fit confidently get file matter wrong because e only read half. Raise am with OLLAMA_CONTEXT_LENGTH for the daemon, or with PARAMETER num_ctx for a Modelfile. If na only one job need the bigger window, you fit set num_ctx per request instead of for the whole server, so the extra cache no go affect every other thing wey daemon dey asked to do. llama.cpp no get any default wey you fit trust either. Set -c explicitly and know wetin you set.

The memory calculation wey nobody dey show you

The model file no be the complete cost. The KV cache (key/value cache) dey keep one entry for each layer and each context token, and e dey grow as conversation dey grow.

Calculate am for Llama 3.1 8B. The model get 32 layers, 8 key/value heads, and head dimension na 128. Each token dey store one key and one value, each one dey use 2 bytes for f16. So, 2 x 8 x 128 x 2 = 4096 bytes for each layer. Across 32 layers, na 128 KiB for each token. Context of 4096 tokens therefore dey use 512 MiB, while context of 32,768 tokens dey use 4 GiB.

So, Q4_K_M 8B model for 4k context need roughly 4.58 GiB for weights, plus about 0.5 GiB for cache, plus the runtime itself. E no go fit inside 4 GiB RAM. E go fit for 8 GiB with some space to work. If you raise the context to 32k for that same 8 GiB machine, the cache alone go finish the available headroom. Monitor am live with free -h while the model dey loaded, and no trust estimate wey you never measure. If you dey size system for something far pass 8B, the same calculation wey dem work through for 27B model for CPU-only VPS shows wetin each tier from 8 to 64 GB fit hold.

Ollama dey multiply this. OLLAMA_NUM_PARALLEL default value na 1, and the memory wey model need dey scale according to that number multiplied by the context length. If you raise both at the same time, the daemon fit quietly request RAM wey be several times more than wetin you expect. That same calculation set your limit for simultaneous users, because each concurrent request need its own part of KV cache. Na this be why server wey dey work fine for one person dey stall when five people use am.

Axis 2: the daemon wey you need operate

The Ollama install script dey write a systemd unit, create one ollama system user, and enable the service. You get lifecycle management without writing any of am yourself. Configuration dey pass through systemd:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -e -u ollama

OLLAMA_KEEP_ALIVE dey matter more for CPU VPS than anywhere else. By default, models dey stay for memory for 5 minutes, then dem unload. The next request must load the whole file again before e fit answer. If kernel still dey hold the file for page cache, e quick, but na spare RAM dey keep am there. Once other work push am out, the file go come back from disk. For slow storage, reload of 4.58 GiB fit turn reply wey take two seconds to one wey take thirty seconds. Long keep-alive go fix the latency, but e go permanently use the RAM. Both options get real cost. Pick the one wey cause less problem. If you decide say the model suppose stay resident, set keep_alive so e survive idle periods and reboots need only a few lines, and e save you from warming the model by hand every time the box restart.

llama.cpp no get daemon or service user, so na you go set up both. From the llama.cpp directory wey you build inside, install the binary and its libraries into /usr/local, create one llama user, and give am one copy of the model:

sudo cmake --install build
sudo ldconfig
sudo useradd --system --no-create-home --shell /usr/sbin/nologin llama
sudo install -d -o llama -g llama /srv/models
sudo install -o llama -g llama -m 644 ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf /srv/models/model-Q4_K_M.gguf

If you use the release archive instead of building, point ExecStart to the llama-server inside am, because the archive keep its libraries beside the binary. Then write the unit as /etc/systemd/system/llama-server.service:

[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
ExecStart=/usr/local/bin/llama-server -m /srv/models/model-Q4_K_M.gguf -c 4096 -t 4 --host 127.0.0.1 --port 8080
Restart=always
RestartSec=3
User=llama

[Install]
WantedBy=multi-user.target

Enable am with sudo systemctl enable --now llama-server. The process go hold the model for the whole time wey e dey run. Nothing go unload when e idle by default, but adding --sleep-idle-seconds go enable automatic unloading and reloading without stopping the service. If writing units be new thing for you, na the same pattern as running your own services under systemd for VPS.

Axis 3: the API wey your app go talk to

This axis don narrow well-well. Both projects now dey use OpenAI chat format, so most client libraries fit work with either one after you change only the base URL.

Ollama dey listen for 127.0.0.1:11434. E get OpenAI-compatible route for http://localhost:11434/v1/chat/completions, and e still keep native API for /api/chat alongside am. Dem document Anthropic-compatible route too. None of these routes allow the model make e look up anything by itself, so if your app need answers wey newer pass the model training data, plan to wire web search through Ollama's API, Open WebUI or one short tool loop.

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama31-q4", "messages": [{"role": "user", "content": "Say this is a test"}]}'

When you start am with the flags above, llama-server dey listen on 127.0.0.1:8080. If you leave am like that, e now default to port 9931 (checked 27 October 2026), so keep --port for the command line if your app dey expect 8080. E serve /v1/chat/completions, /v1/completions and /v1/embeddings, plus e own /completion endpoint and built-in web UI. E also expose operational routes wey Ollama no get: /health for readiness probe, /props for the loaded model settings, /slots for wetin each request slot dey do, and /metrics for Prometheus format after you add --metrics, because that endpoint dey off by default. If you plan to monitor this service, na this difference fit decide am.

Neither server dey switch on authentication for you. Both default to loopback for good reason. Reach dem through an SSH tunnel or from behind a reverse proxy, and never open 11434 or 8080 to the internet.

Wetin CPU-only VPS fit honestly do

CPU-only VPS dey run small models slowly. Na the honest summary be that, and the useful part na to know where the limit dey. Measure first before you design anything around am:

llama-bench -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128

The pp column na prompt processing speed, while the tg column na token generation speed. Both dey measure tokens per second. For shared vCPU plan, 8B model with Q4_K_M usually dey reach low single digits for tg. Prompt processing na the part wey dey cause most delay: the system go process the whole prompt before the first output token show, so long system prompt go add wait to every request. Reply length na the part of the bill wey you fit control, because at three tokens per second, model wey dey ramble for 600 tokens go hold the box for 200 seconds, more than three minutes of generation alone. So capping the output with num_predict na the cheapest way to stop one talkative answer from turning into timeout.

CPU fit handle: 1B to 4B model wey dey do classification, extraction, short summaries, or routing. Replies go show within seconds, and the memory go fit normal plan. For worked example at that size instead of size range, Nemotron 3.5 Lightning pulled and measured on a VPS gives the exact tag, the RAM wey e actually need, and the speed wey e maintain without GPU. CPU no fit handle well: interactive chat at reading speed, coding assistants, long-document work, or anything with agent loop wey dey make many calls one after another. Loop wey make twelve calls at four seconds each go take one minute before e produce anything. If coding assistant na still the plan, pointing an agent at a model you host yourself explains which jobs small local model genuinely fit do better and which ones must remain on hosted API.

Two ways dey out when the numbers no work. If na concurrency be the problem, meaning many users dey hit one model at the same time, engine choice go change, and the comparison of Ollama against vLLM for concurrent serving covers that area. If na raw speed be the problem, the answer na a VPS with a GPU attached, where -ngl starts to matter. Before you choose either one, get baseline for the hardware itself, because disk and memory bandwidth affect load time almost as much as CPU. A repeatable VPS benchmark worth the one hour.

Install llama.cpp, wey you pin to one build

Both projects dey change every week, so record the version wey you deploy. The upstream one-liner dey install the current build:

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

To pin one specific build, collect the prebuilt tarball from the releases page instead. The project now dey publish semantic-versioned releases too: v0.7.0 na the latest one for 27 October 2026, and the release page point to nightly b11669 for the binaries. v0.8.0 follow on 28 October 2026, and e release page point to nightly b11960 for the binaries. The commands below use b10224, wey be the current tag for 2 August 2026; replace am with any tag wey you don test:

curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10224/llama-b10224-bin-ubuntu-x64.tar.gz
tar xf llama-b10224-bin-ubuntu-x64.tar.gz
find . -type f -name 'llama-server'

Or build that same tag from source:

sudo apt update && sudo apt install -y build-essential cmake git libssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10224
cmake -B build
cmake --build build --config Release -j $(nproc)

libssl-dev na the documented dependency for the HTTPS features. The compile fit take several minutes and e need more RAM than the smallest plans get, so build am for a bigger box and copy the binaries if the small one run out of resources.

Install Ollama, pin am to one version

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.32.5 sh
ollama -v

The script dey read OLLAMA_VERSION, so you fit hold one release wey you don verify instead of taking anything wey release this morning. v0.32.5 come out on 27 July 2026. Manual option dey too if you no wan pipe script enter shell:

sudo rm -rf /usr/lib/ollama
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr
ollama -v

The manual method no dey create systemd unit or service user, so na you go add dem. The complete Ollama for VPS walkthrough explain the service setup step by step.

Problems wey fit happen, and the messages wey you go see

Ollama no gree load the model. Older Ollama releases dey check the size before loading, and ollama run dey return one line like this:

Error: model requires more system memory (5.6 GiB) than is available (3.2 GiB)

Current Ollama source no longer get that message (checked 27 October 2026). Ollama now hand loading over to llama-server, so when model no fit enter memory, e go show as runner failure instead, with errors wey start like these:

llama-server reported out-of-memory during startup
llama-server process has terminated

Either way, na the same fix. Move down one quantisation row, reduce the context length, or choose a smaller model.

llama.cpp no fail, e just crawl. llama.cpp dey memory-map the GGUF by default, so file wey pass RAM size fit still start. Kernel go then page weights in and out from disk for every token, and generation go drop to seconds per token while disk dey pinned at 100 percent. Pass --load-mode none to disable memory mapping. That option replace --no-mmap, wey pinned b10224 build document as deprecated and current builds no longer get (checked 29 October 2026). When kernel step in, dmesg go show the reason:

Out of memory: Killed process 1234 (llama-server)

The model file no go load at all. GGUF wey you build for model family wey newer than your engine go return error wey name the architecture e no recognise, where <arch> na the name stored for the file metadata:

llama_model_load: error loading model: unknown model architecture: '<arch>'

The fix na engine upgrade, no be different file. Na the price of pinning, and na why you write down the build number. You need know the version wey you dey upgrade from.

The API dey answer locally but e no answer from your app. Ollama dey bind 127.0.0.1:11434, so another host go get connection refused. Set OLLAMA_HOST=0.0.0.0:11434 through systemctl edit ollama only when port dey behind firewall or private network, because API no get authentication in front of am.

The first reply after pause dey very slow. The 5 minute idle unload don happen, and model dey load again, from disk if page cache no keep am. ollama ps wey you run just before request no go show anything loaded, and this confirm am. Increase OLLAMA_KEEP_ALIVE.

So which one you suppose run?

Run Ollama when you want make e manage models for you and provide an OpenAI-shaped endpoint without extra work. E be the correct default for first deployment, and for anything wey model choice go continue to change.

Run llama.cpp directly when memory tight reach where you need choose the quantisation row yourself, when you want /health, /slots and /metrics for monitoring, or when you need a flag wey Ollama no expose. E be the honest choice for VPS where model barely fit, because na the same settings wey make am fit be exactly the ones wey Ollama choose for you.

Running both dey normal. Ollama for experiments, llama.cpp for the one model wey you put for production and never want make e move.

FAQ

Ollama na just wrapper around llama.cpp?

Close, but the wrapper dey do real work. Ollama's README list llama.cpp as its inference backend (checked 27 October 2026). On top of am, Ollama add model registry, prompt template wey dey turn chat messages into prompt, set of default sampling parameters, daemon with idle unloading, and HTTP API. When you compare tokens per second with identical settings, you dey compare the same engine with itself. Wetin you actually dey choose between na the management layer.

Which one dey faster for CPU-only VPS?

Dem share the same engine, so with the same model file, quantisation, context size and thread count, dem dey perform almost the same. Differences wey people report usually come from different defaults, mostly context length and thread count, no be from the engine. Measure am with llama-bench -m <file> -p 512 -n 128 and compare the tg column for your own box before you believe any published figure.

I fit use my own GGUF file with Ollama?

Yes. Put the file for the server, write a Modelfile wey first line na FROM ./your-model.gguf, add any PARAMETER lines wey you need, like num_ctx, then run ollama create your-name -f ./Modelfile. ollama ls go list am beside anything wey you pull from the registry. Na so you fit use quantisation wey the registry no get.

How much RAM I need for an 8B model?

Plan for the file size, plus the KV cache, plus the runtime. A Q4_K_M build of Llama 3.1 8B dey around 4.58 GiB for disk, and 4096 token context dey add roughly 512 MiB cache, so 8 GiB RAM dey comfortable, while 4 GiB no reach. The cache dey scale with context: the same model with 32,768 token context need about 4 GiB cache by itself. With Ollama, remember say the requirement still dey scale with OLLAMA_NUM_PARALLEL.