Run the llama.cpp server on a VPS
Build llama.cpp from a pinned tag, serve GGUF models with OpenAI API compatibility, bind it to localhost, and run it under systemd with memory limits.
What you are building
Running the llama.cpp server on a VPS means one binary, llama-server, that loads a single GGUF model file and answers HTTP requests on an OpenAI-compatible API. Point a compatible client at http://127.0.0.1:8080/v1, then test the features your application uses: compatibility is partial. The install is the easy half.
The rest of the work is operations: pin a version, keep the port on localhost, write a systemd unit, and decide what happens when the box runs out of memory. That is what this guide covers. If you have not yet decided between the two obvious options, read the trade-offs between Ollama and llama.cpp first, because this is the how-to that comparison leaves out on purpose.
Pick a release tag and write it down
llama.cpp uses build-number tags such as b10488 for nightly development builds and semantic version tags, vX.Y.Z, for official releases. b10488 is the pinned build used here, released on 18 August 2026. "Latest" is a moving target and the version you tested is the only version you can support. Pick a tag, record it, and use that same string in your clone, in your binary name and in your notes.
The b10488 tag also ships prebuilt archives. For a CPU-only x86 VPS that is llama-b10488-bin-ubuntu-x64.tar.gz, and an arm64 archive sits next to it if you are on an ARM VPS rather than x86.
curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10488/llama-b10488-bin-ubuntu-x64.tar.gz
tar tf llama-b10488-bin-ubuntu-x64.tar.gz | headList the archive before extracting it, so you know where the files land. Those binaries are linked against the C library of the image that built them, so on an older distribution they fail at startup with an error naming a GLIBC_ version that is not installed. Building from source takes a few minutes on a small VPS and removes that whole class of problem, so that is the path below.
Build llama-server from a pinned tag
sudo apt update
sudo apt install -y build-essential cmake git libssl-dev
git clone --depth 1 --branch b10488 https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --config Release -t llama-server -j 2--branch b10488 on a --depth 1 clone checks out that tag and nothing else, so the build cannot drift while you are working.
libssl-dev matters because the LLAMA_OPENSSL option is on by default, and that is what lets the binary download models over HTTPS later. Without OpenSSL development files, this pinned build warns that HTTPS support is disabled instead of failing configuration.
-DBUILD_SHARED_LIBS=OFF links llama.cpp and GGML into the executable; system library dependencies remain. The default build puts shared libraries next to the executable, so copying only the executable to /usr/local/bin then fails with error while loading shared libraries: libllama.so.
-t llama-server builds only the server target. The default build also compiles the other tools and the tests, which on a two core VPS is several extra minutes spent on files you will never run.
-j 2 is deliberate. Each parallel compile job holds its own working set, so -j $(nproc) on a small plan ends with c++: fatal error: Killed signal terminated program cc1plus, which is the kernel out-of-memory killer stopping the compiler. Lower the job count, or add swap for the build.
One flag you may want to change: GGML_NATIVE defaults to on, so the compiler targets the exact CPU doing the build. That is what you want when you build on the machine that will run it. If you build once and copy the binary to a different host, add -DGGML_NATIVE=OFF, because a binary using instructions the other CPU does not have dies with Illegal instruction (core dumped) on the first inference.
Install it under a name that carries the tag.
./build/bin/llama-server --version
sudo install -m 755 build/bin/llama-server /usr/local/bin/llama-server-b10488
sudo ln -sfn /usr/local/bin/llama-server-b10488 /usr/local/bin/llama-server--version prints the build number and commit. It has to match the tag you checked out. If it does not, you built something else. Keeping the number in the filename and pointing a symlink at it means an upgrade is one ln -sfn plus one restart, and a rollback is the same command with the old number.
Get a GGUF model, and check the disk first
GGUF is the single file format llama.cpp loads. One file holds the weights, the tokeniser and the metadata, so there is nothing else to install. The suffix on the filename is the quantisation, which is the precision the weights are stored at: Q4_K_M is a 4-bit mix, Q8_0 is 8-bit, and f16 is the unquantised half-precision file.
Create a service account and a model directory before you download anything.
sudo useradd --system --home /srv/llama --create-home --shell /usr/sbin/nologin llama
sudo install -d -o llama -g llama /srv/models
df -h /srvThe server can fetch a model itself with -hf, which is the fastest way to prove your build works.
sudo -u llama env LLAMA_CACHE=/srv/models /usr/local/bin/llama-server \
-hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M --host 127.0.0.1 --port 8080LLAMA_CACHE sets the download directory. Without it or XDG_CACHE_HOME, the file goes to ~/.cache/llama.cpp under whichever account ran the command; the unit below makes /srv/llama read-only. Run ls -lh /srv/models afterwards, because the cached filename is derived from the repository name rather than from the plain filename.
For a service, download to a path you chose, so the unit file has something stable to point at.
sudo -u llama curl -L --output-dir /srv/models -O \
https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.ggufDisk is the limit people meet first. These are the published file sizes for two models, checked on 18 August 2026.
The data behind this chart
[
{
"label": "gemma-3-1b-it Q4_K_M",
"size_gb": 0.81
},
{
"label": "gemma-3-1b-it Q8_0",
"size_gb": 1.07
},
{
"label": "gemma-3-1b-it f16",
"size_gb": 2.01
},
{
"label": "gpt-oss-20b MXFP4",
"size_gb": 12.11
}
]The 4-bit file for the 1B model is 0.81 GB. The same model with no quantisation is 2.01 GB, so the format choice moves the number by more than a factor of two. A 20B model at MXFP4 is 12.11 GB, and it still has to be read into memory after that. If you are eyeing a specific family, the same sizing exercise for GLM shows how quickly the headline model prices itself out of a VPS while a smaller sibling fits.
Check df -h before every download. A root filesystem that fills up during a 12 GB transfer breaks everything else that wants to write, including the journal.
Run it once by hand, and check it
sudo -u llama /usr/local/bin/llama-server \
--model /srv/models/gemma-3-1b-it-Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 \
--ctx-size 4096 --parallel 1 --threads 2 --no-webuiIn a second session, ask the server whether it is ready.
curl -s http://127.0.0.1:8080/healthWhile the file is loading you get HTTP 503 and this body:
{"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}When it is ready the body is {"status": "ok" }. Then send a real request.
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"local","messages":[{"role":"user","content":"Say hello in five words."}]}'A JSON object with a choices array is a working server. The model field is there because OpenAI clients always send one. This server has a single model loaded, so the value is not used to select anything.
The OpenAI-compatible API, and what else is on the port
POST /v1/chat/completions, POST /v1/completions, POST /v1/responses and POST /v1/embeddings are OpenAI-compatible routes, and GET /v1/models reports the loaded model. Embeddings require a model with pooling other than none. GET /health is the readiness check above, GET /props returns the server's current settings, and GET /metrics exposes Prometheus counters when you start with --metrics.
For compatible OpenAI SDK calls, set the base URL to http://127.0.0.1:8080/v1 and supply a non-empty API key string if the SDK requires one. This does not guarantee compatibility with every SDK feature. Nothing checks that key until you set --api-key or --api-key-file yourself.
Do not take anyone's throughput claim as a number for your own plan. CPU inference speed depends on the core count, the memory bandwidth and the neighbours you share the host with, so measure tokens per second on your own box and treat that result as the truth. Steal time from a noisy neighbour shows up here as generation speed that changes from hour to hour.
Keep it on 127.0.0.1 and put a proxy in front
--host already defaults to 127.0.0.1, so the server is unreachable from outside until you change it. Leave it alone. Use the proxy for user accounts, request rate limits and audit logging; built-in API authentication uses static keys: --api-key takes one key or a comma-separated list, and --api-key-file reads one key per line. An open inference port is free compute for whoever finds it, and the same mistake made with Ollama has the same shape: locking down a self-hosted model API applies here line for line.
Terminate TLS (transport layer security) in nginx and proxy to the loopback port.
server {
listen 443 ssl;
server_name llm.example.com;
location /v1/ {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_read_timeout 600s;
}
}proxy_buffering off passes streaming responses to the client as nginx receives them. With buffering on, nginx buffers upstream data instead of forwarding each event immediately. proxy_read_timeout 600s gives slow answers room. It is the longest nginx waits between two reads from the server, not a cap on the whole response, so with the default of 60 seconds a non-streaming request that sends nothing for that interval times out. In b10488, streaming responses send SSE keepalive pings every 30 seconds by default, including during long prompt processing. Get the certificate with Certbot and Let's Encrypt on nginx.
The systemd unit
Write /etc/systemd/system/llama-server.service.
[Unit]
Description=llama.cpp server
After=network-online.target
Wants=network-online.target
[Service]
User=llama
Group=llama
Environment=LLAMA_ARG_MODEL=/srv/models/gemma-3-1b-it-Q4_K_M.gguf
Environment=LLAMA_ARG_HOST=127.0.0.1
Environment=LLAMA_ARG_PORT=8080
Environment=LLAMA_ARG_CTX_SIZE=4096
Environment=LLAMA_ARG_N_PARALLEL=1
Environment=LLAMA_ARG_THREADS=2
ExecStart=/usr/local/bin/llama-server --no-webui
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
MemoryHigh=3G
MemoryMax=3500M
OOMPolicy=stop
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
[Install]
WantedBy=multi-user.targetThe settings live in Environment= lines because llama-server reads LLAMA_ARG_* variables for most flags, and a command line argument overrides the matching variable. That gives you one place to change the context size, and it keeps ExecStart short enough to read at a glance.
ProtectSystem=strict makes the filesystem read-only for this unit, with exceptions including the private temporary directories, which is fine because the server only reads the model. Add ReadWritePaths=/srv/models and Environment=LLAMA_CACHE=/srv/models if you want the service itself to download models with -hf. ProtectHome=yes hides /home, /root and /run/user; it does not hide this service account's home at /srv/llama, but ProtectSystem=strict makes that directory read-only.
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
systemctl status llama-server
curl -s http://127.0.0.1:8080/health
journalctl -u llama-server -n 50 --no-pagerenable --now is the half people skip. Without enable, the server is gone after the next reboot. If you want scheduled work around the service, such as a nightly check for a new release, a systemd service plus timer is the mechanism for it.
Decide what happens on OOM before it happens
Memory use has two parts, and they behave differently under a limit. The model file is memory-mapped by default, so its pages are file-backed: the kernel can drop them and read them again from disk. The KV cache, which is the per-token state the server keeps for every active conversation, is anonymous memory. It cannot be dropped, so it is what gets the process killed.
That is why the two limits in the unit do different jobs. MemoryHigh=3G is a soft limit: above it the kernel puts the cgroup under reclaim pressure, so mapped model pages get evicted and are read back from disk on the next token. The service keeps working and gets slower. MemoryMax=3500M is the hard ceiling: the kernel reclaims what it can first, and if usage still cannot be held under the limit the OOM killer runs inside the unit, and the journal says so plainly.
llama-server.service: A process of this unit has been killed by the OOM killer.Set --ctx-size yourself. The default is 0, which means the context the model was trained with, and on a modern long-context model that allocates a very large KV cache at startup. The service then dies before it serves a single request. --parallel does not multiply that cost on its own: --ctx-size sets the total size of the KV cache, shared by every slot, so --ctx-size 4096 --parallel 4 gives each conversation about 1,024 tokens rather than four times the memory. Giving every slot the full context means multiplying --ctx-size by the slot count, and the memory grows with it, so leave --parallel at 1 until you know you need concurrency.
With Restart=on-failure a killed service comes back. Systemd stops restarting it with start request repeated too quickly only when starts exceed the configured rate limit; the five-second restart delay here does not reach the default limit of five starts in ten seconds. Stop a persistent restart loop with sudo systemctl stop llama-server, fix the limit or the context size, then clear any failed state with sudo systemctl reset-failed llama-server and start it again.
Watch the real number with systemctl show llama-server -p MemoryCurrent while a request is running. Capping process memory and CPU with systemd covers these directives in more detail.
Avoid swap for this workload. A swapped-out model turns every token into random-offset disk reads. Memory mapping the model file achieves the same effect with less harm, because the kernel reads the pages it needs straight from the file.
Where Ollama is the better answer
This is a fork in the road. Choose llama-server when you want one process with flags you set, a build you pinned, and a file you chose, where nothing changes underneath you because nothing else is running.
Choose Ollama when you want model management: pulling models by name, keeping several on disk, unloading an idle one, and upgrading with a single command instead of a rebuild. That is real work you would otherwise script yourself. Running Ollama on a VPS is this same job with the trade made the other way. Both serve an OpenAI-compatible API, but test the endpoints and features your client uses before switching either direction.
Upgrading a pinned build
Replace bNNNNN with the nightly build tag you are moving to; official release tags use vX.Y.Z instead.
cd llama.cpp
git fetch --tags
git checkout bNNNNN
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --config Release -t llama-server -j 2
sudo install -m 755 build/bin/llama-server /usr/local/bin/llama-server-bNNNNN
sudo ln -sfn /usr/local/bin/llama-server-bNNNNN /usr/local/bin/llama-server
sudo systemctl restart llama-serverThe old binary stays on disk, so a rollback is one ln -sfn back to llama-server-b10488 and one restart. Read the release notes before you move. GGUF files are versioned, but backward compatibility is not universal: the current loader rejects GGUFv1. Flags also get renamed: in b10488, --mlock and --no-mmap were already deprecated in favour of --load-mode, and as of 11 October 2026 the current server documentation no longer lists them, so a unit file that passes a removed flag fails at start with an unrecognised argument message. Defaults move too: current builds listen on port 9931 by default, where b10488 used 8080, which is why every command and the unit here set the port explicitly.
Failure modes, and the strings you will see
error while loading shared libraries: libllama.so after you copy the binary somewhere. The default build produces shared libraries alongside it. Rebuild with -DBUILD_SHARED_LIBS=OFF, or copy the whole build/bin directory.
Illegal instruction (core dumped) at startup or on the first request. The binary was compiled with GGML_NATIVE on, for a different CPU than the one running it. Rebuild on this machine, or configure with -DGGML_NATIVE=OFF.
c++: fatal error: Killed signal terminated program cc1plus during the build. The compiler was killed for using too much memory. Lower -j, or add swap for the build and remove it afterwards.
curl: (7) Failed to connect ... Connection refused from your laptop. That is correct: the server listens on the loopback address of the VPS. Test on the VPS itself, or open a tunnel with ssh -L 8080:127.0.0.1:8080 user@your-vps and use http://127.0.0.1:8080 locally.
HTTP 503 with "message":"Loading model" for the first seconds or minutes after a restart. Reading a multi-gigabyte file takes time, and systemd reports the unit as active the moment the process starts, well before the model is in memory.
Requests hang and then return 504 Gateway Time-out. The server sent nothing for longer than proxy_read_timeout, which can happen on non-streaming requests. Streaming requests in b10488 send keepalive pings every 30 seconds by default during long prompt processing. Raise proxy_read_timeout, and turn off proxy_buffering so tokens reach the client as they are produced.
The unit flaps and then stops with start request repeated too quickly. Its starts exceeded the configured rate limit. Check journalctl -u llama-server for the OOM killer line, then lower --ctx-size or raise MemoryMax.
FAQ
Should I run llama.cpp's server or Ollama on my VPS?
Run llama-server when you want to pin an exact build, pass exact flags, and keep one model in one file that nothing updates behind your back. Run Ollama when you want model management and one-command upgrades, because pulling models by name, keeping several on disk and unloading idle ones is work you would otherwise script yourself. Both serve an OpenAI-compatible API, but switching requires checking the endpoints and features your client uses.
Which llama.cpp version should I pin to?
Any tag you have actually built and tested. Nightly development builds use build-number tags such as b10488, released on 18 August 2026; official releases use semantic version tags, vX.Y.Z. Clone with --branch <tag>, install the binary under a filename containing that tag, and point a symlink at it, so upgrade and rollback are one command each.
How much RAM does llama-server need?
Start from the size of the GGUF file, then add the KV cache, which grows with --ctx-size. That figure is the total token budget shared by all --parallel slots, so more slots need a larger --ctx-size if each one is to keep the same context. Published figures are not a substitute for measuring your own setup, because the total depends on the model, the quantisation and the context you allow. Run systemctl show llama-server -p MemoryCurrent while a request is in flight and use the number you see.
Why does /health return 503 with "Loading model"?
The process has started but the model file is not in memory yet, so the server answers {"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}. This is normal after every restart, and it lasts as long as reading the file takes. It becomes a problem only when a client or a proxy treats that first 503 as a hard failure. Poll /health until it returns {"status": "ok" }.
Can I expose llama-server directly to the internet?
Do not bind it to 0.0.0.0 and open the port. Use the proxy for accounts, request rate limits and audit logging; built-in API authentication uses static keys set with --api-key or --api-key-file. Keep the default 127.0.0.1 bind, put nginx in front with TLS, and set --api-key as well, so one mistake in the proxy config does not leave the model open to everyone.