Run the llama.cpp server on a VPS

What you are building

Running the llama.cpp server on a VPS means one binary, llama-server, that loads a single GGUF model file and answers HTTP requests on an OpenAI-compatible API. Point any OpenAI client at http://127.0.0.1:8080/v1 and it works. The install is the easy half.

The rest of the work is operations: pin a version, keep the port on localhost, write a systemd unit, and decide what happens when the box runs out of memory. That is what this guide covers. If you have not yet decided between the two obvious options, read the trade-offs between Ollama and llama.cpp first, because this is the how-to that comparison leaves out on purpose.

Pick a release tag and write it down

llama.cpp tags a release for nearly every merge, so the tags are build numbers. b10488 is the newest as of 18 August 2026. There is no long-lived stable branch, which means "latest" is a moving target and the version you tested is the only version you can support. Pick a tag, record it, and use that same string in your clone, in your binary name and in your notes.

Each tag also ships prebuilt archives. For a CPU-only x86 VPS that is llama-b10488-bin-ubuntu-x64.tar.gz, and an arm64 archive sits next to it if you are on an ARM VPS rather than x86.

curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b10488/llama-b10488-bin-ubuntu-x64.tar.gz
tar tf llama-b10488-bin-ubuntu-x64.tar.gz | head

List the archive before extracting it, so you know where the files land. Those binaries are linked against the C library of the image that built them, so on an older distribution they fail at startup with an error naming a GLIBC_ version that is not installed. Building from source takes a few minutes on a small VPS and removes that whole class of problem, so that is the path below.

Build llama-server from a pinned tag

sudo apt update
sudo apt install -y build-essential cmake git libssl-dev
git clone --depth 1 --branch b10488 https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --config Release -t llama-server -j 2

--branch b10488 on a --depth 1 clone checks out that tag and nothing else, so the build cannot drift while you are working.

libssl-dev matters because the LLAMA_OPENSSL option is on by default, and that is what lets the binary download models over HTTPS later. Without the headers the configure step fails.

-DBUILD_SHARED_LIBS=OFF gives you one self-contained binary. The default build puts shared libraries next to the executable, so copying only the executable to /usr/local/bin then fails with error while loading shared libraries: libllama.so.

-t llama-server builds only the server target. The default build also compiles the other tools and the tests, which on a two core VPS is several extra minutes spent on files you will never run.

-j 2 is deliberate. Each parallel compile job holds its own working set, so -j $(nproc) on a small plan ends with c++: fatal error: Killed signal terminated program cc1plus, which is the kernel out-of-memory killer stopping the compiler. Lower the job count, or add swap for the build.

One flag you may want to change: GGML_NATIVE defaults to on, so the compiler targets the exact CPU doing the build. That is what you want when you build on the machine that will run it. If you build once and copy the binary to a different host, add -DGGML_NATIVE=OFF, because a binary using instructions the other CPU does not have dies with Illegal instruction (core dumped) on the first inference.

Install it under a name that carries the tag.

./build/bin/llama-server --version
sudo install -m 755 build/bin/llama-server /usr/local/bin/llama-server-b10488
sudo ln -sfn /usr/local/bin/llama-server-b10488 /usr/local/bin/llama-server

--version prints the build number and commit. It has to match the tag you checked out. If it does not, you built something else. Keeping the number in the filename and pointing a symlink at it means an upgrade is one ln -sfn plus one restart, and a rollback is the same command with the old number.

Get a GGUF model, and check the disk first

GGUF is the single file format llama.cpp loads. One file holds the weights, the tokeniser and the metadata, so there is nothing else to install. The suffix on the filename is the quantisation, which is the precision the weights are stored at: Q4_K_M is a 4-bit mix, Q8_0 is 8-bit, and f16 is the unquantised half-precision file.

Create a service account and a model directory before you download anything.

sudo useradd --system --home /srv/llama --create-home --shell /usr/sbin/nologin llama
sudo install -d -o llama -g llama /srv/models
df -h /srv

The server can fetch a model itself with -hf, which is the fastest way to prove your build works.

sudo -u llama env LLAMA_CACHE=/srv/models /usr/local/bin/llama-server \
  -hf ggml-org/gemma-3-1b-it-GGUF:Q4_K_M --host 127.0.0.1 --port 8080

LLAMA_CACHE sets the download directory. Without it the file goes to ~/.cache/llama.cpp under whichever account ran the command, which is the wrong place for a service whose home directory you are about to make unreadable. Run ls -lh /srv/models afterwards, because the cached filename is derived from the repository name rather than from the plain filename.

For a service, download to a path you chose, so the unit file has something stable to point at.

sudo -u llama curl -L --output-dir /srv/models -O \
  https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf

Disk is the limit people meet first. These are the published file sizes for two models, checked on 18 August 2026.

[
  {
    "label": "gemma-3-1b-it Q4_K_M",
    "size_gb": 0.81
  },
  {
    "label": "gemma-3-1b-it Q8_0",
    "size_gb": 1.07
  },
  {
    "label": "gemma-3-1b-it f16",
    "size_gb": 2.01
  },
  {
    "label": "gpt-oss-20b MXFP4",
    "size_gb": 12.11
  }
]

The 4-bit file for the 1B model is 0.81 GB. The same model with no quantisation is 2.01 GB, so the format choice moves the number by more than a factor of two. A 20B model at MXFP4 is 12.11 GB, which does not fit the disk on many entry-level plans, and it still has to be read into memory after that.

Check df -h before every download. A root filesystem that fills up during a 12 GB transfer breaks everything else that wants to write, including the journal.

Run it once by hand, and check it

sudo -u llama /usr/local/bin/llama-server \
  --model /srv/models/gemma-3-1b-it-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 4096 --parallel 1 --threads 2 --no-webui

In a second session, ask the server whether it is ready.

curl -s http://127.0.0.1:8080/health

While the file is loading you get HTTP 503 and this body:

{"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}

When it is ready the body is {"status": "ok" }. Then send a real request.

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Say hello in five words."}]}'

A JSON object with a choices array is a working server. The model field is there because OpenAI clients always send one. This server has a single model loaded, so the value is not used to select anything.

The OpenAI-compatible API, and what else is on the port

POST /v1/chat/completions, POST /v1/completions and POST /v1/embeddings are the OpenAI-compatible routes, and GET /v1/models reports the loaded model. GET /health is the readiness check above, GET /props returns the server's current settings, and GET /metrics exposes Prometheus counters when you start with --metrics.

Any OpenAI SDK works once you set the base URL to http://127.0.0.1:8080/v1 and pass a non-empty API key string. Nothing checks that key until you set --api-key yourself.

Do not take anyone's throughput claim as a number for your own plan. CPU inference speed depends on the core count, the memory bandwidth and the neighbours you share the host with, so measure tokens per second on your own box and treat that result as the truth. Steal time from a noisy neighbour shows up here as generation speed that changes from hour to hour.

Keep it on 127.0.0.1 and put a proxy in front

--host already defaults to 127.0.0.1, so the server is unreachable from outside until you change it. Leave it alone. There is no user model, no rate limit and no useful audit log in llama-server, and the single built-in control is --api-key, which compares one string. An open inference port is free compute for whoever finds it, and the same mistake made with Ollama has the same shape: locking down a self-hosted model API applies here line for line.

Terminate TLS (transport layer security) in nginx and proxy to the loopback port.

server {
    listen 443 ssl;
    server_name llm.example.com;

    location /v1/ {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
}

proxy_buffering off is required for streaming. With buffering on, nginx holds the server-sent events (SSE) until the response finishes, so the client waits in silence and then receives the whole answer at once. proxy_read_timeout 600s covers long generations, because the default of 60 seconds turns a slow answer into 504 Gateway Time-out. Get the certificate with Certbot and Let's Encrypt on nginx.

The systemd unit

Write /etc/systemd/system/llama-server.service.

[Unit]
Description=llama.cpp server
After=network-online.target
Wants=network-online.target

[Service]
User=llama
Group=llama
Environment=LLAMA_ARG_MODEL=/srv/models/gemma-3-1b-it-Q4_K_M.gguf
Environment=LLAMA_ARG_HOST=127.0.0.1
Environment=LLAMA_ARG_PORT=8080
Environment=LLAMA_ARG_CTX_SIZE=4096
Environment=LLAMA_ARG_N_PARALLEL=1
Environment=LLAMA_ARG_THREADS=2
ExecStart=/usr/local/bin/llama-server --no-webui
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
MemoryHigh=3G
MemoryMax=3500M
OOMPolicy=stop
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes

[Install]
WantedBy=multi-user.target

The settings live in Environment= lines because llama-server reads LLAMA_ARG_* variables for most flags, and a command line argument overrides the matching variable. That gives you one place to change the context size, and it keeps ExecStart short enough to read at a glance.

ProtectSystem=strict makes the whole filesystem read-only for this unit, which is fine because the server only reads the model. Add ReadWritePaths=/srv/models if you want the service itself to download models with -hf. ProtectHome=yes hides /home and /root, and that is the second reason to keep models in /srv: with ProtectHome on, the default ~/.cache/llama.cpp path is not visible to the process at all.

sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
systemctl status llama-server
curl -s http://127.0.0.1:8080/health
journalctl -u llama-server -n 50 --no-pager

enable --now is the half people skip. Without enable, the server is gone after the next reboot. If you want scheduled work around the service, such as a nightly check for a new release, a systemd service plus timer is the mechanism for it.

Decide what happens on OOM before it happens

Memory use has two parts, and they behave differently under a limit. The model file is memory-mapped by default, so its pages are file-backed: the kernel can drop them and read them again from disk. The KV cache, which is the per-token state the server keeps for every active conversation, is anonymous memory. It cannot be dropped, so it is what gets the process killed.

That is why the two limits in the unit do different jobs. MemoryHigh=3G is a soft limit: above it the kernel puts the cgroup under reclaim pressure, so mapped model pages get evicted and are read back from disk on the next token. The service keeps working and gets slower. MemoryMax=3500M is a hard limit: above that the process is killed, and the journal says so plainly.

llama-server.service: A process of this unit has been killed by the OOM killer.

Set --ctx-size yourself. The default is 0, which means the context the model was trained with, and on a modern long-context model that allocates a very large KV cache at startup. The service then dies before it serves a single request. --parallel multiplies the same cost, because each slot holds its own conversation state, so leave it at 1 until you know you need concurrency.

With Restart=on-failure a killed service comes back. If it is killed on every start, systemd gives up and systemctl status prints start request repeated too quickly. That is the correct behaviour: a restart loop that re-reads a 12 GB file every five seconds is worse than an outage. Fix the limit or the context size, then clear the state with sudo systemctl reset-failed llama-server.

Watch the real number with systemctl show llama-server -p MemoryCurrent while a request is running. Capping process memory and CPU with systemd covers these directives in more detail.

Avoid swap for this workload. A swapped-out model turns every token into random-offset disk reads. Memory mapping the model file achieves the same effect with less harm, because the kernel reads the pages it needs straight from the file.

Where Ollama is the better answer

This is a fork in the road. Choose llama-server when you want one process with flags you set, a build you pinned, and a file you chose, where nothing changes underneath you because nothing else is running.

Choose Ollama when you want model management: pulling models by name, keeping several on disk, unloading an idle one, and upgrading with a single command instead of a rebuild. That is real work you would otherwise script yourself. Running Ollama on a VPS is this same job with the trade made the other way. Both serve an OpenAI-compatible API, so client code survives the switch either direction.

Upgrading a pinned build

Replace bNNNNN with the tag you are moving to.

cd llama.cpp
git fetch --tags
git checkout bNNNNN
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --config Release -t llama-server -j 2
sudo install -m 755 build/bin/llama-server /usr/local/bin/llama-server-bNNNNN
sudo ln -sfn /usr/local/bin/llama-server-bNNNNN /usr/local/bin/llama-server
sudo systemctl restart llama-server

The old binary stays on disk, so a rollback is one ln -sfn back to llama-server-b10488 and one restart. Read the release notes before you move. GGUF files are versioned and old ones keep loading, but flags do get renamed: --mlock and --no-mmap are already deprecated in favour of --load-mode, and a unit file that passes a removed flag fails at start with an unrecognised argument message.

Failure modes, and the strings you will see

error while loading shared libraries: libllama.so after you copy the binary somewhere. The default build produces shared libraries alongside it. Rebuild with -DBUILD_SHARED_LIBS=OFF, or copy the whole build/bin directory.

Illegal instruction (core dumped) at startup or on the first request. The binary was compiled with GGML_NATIVE on, for a different CPU than the one running it. Rebuild on this machine, or configure with -DGGML_NATIVE=OFF.

c++: fatal error: Killed signal terminated program cc1plus during the build. The compiler was killed for using too much memory. Lower -j, or add swap for the build and remove it afterwards.

curl: (7) Failed to connect ... Connection refused from your laptop. That is correct: the server listens on the loopback address of the VPS. Test on the VPS itself, or open a tunnel with ssh -L 8080:127.0.0.1:8080 user@your-vps and use http://127.0.0.1:8080 locally.

HTTP 503 with "message":"Loading model" for the first seconds or minutes after a restart. Reading a multi-gigabyte file takes time, and systemd reports the unit as active the moment the process starts, well before the model is in memory.

Requests hang and then return 504 Gateway Time-out. The proxy gave up before the model finished. Raise proxy_read_timeout, and turn off proxy_buffering so tokens reach the client as they are produced.

The unit flaps and then stops with start request repeated too quickly. Something kills it on every start. Check journalctl -u llama-server for the OOM killer line, then lower --ctx-size, lower --parallel, or raise MemoryMax.

FAQ

Should I run llama.cpp's server or Ollama on my VPS?

Run llama-server when you want to pin an exact build, pass exact flags, and keep one model in one file that nothing updates behind your back. Run Ollama when you want model management and one-command upgrades, because pulling models by name, keeping several on disk and unloading idle ones is work you would otherwise script yourself. Both serve an OpenAI-compatible API, so client code does not change if you switch later.

Which llama.cpp version should I pin to?

Any tag you have actually built and tested. llama.cpp tags nearly every merge and the names are build numbers such as b10488, which was the newest on 18 August 2026. There is no separate stable branch, so "current" moves several times a day. Clone with --branch <tag>, install the binary under a filename containing that tag, and point a symlink at it, so upgrade and rollback are one command each.

How much RAM does llama-server need?

Start from the size of the GGUF file, then add the KV cache, which grows with --ctx-size and with the number of --parallel slots. Published figures are not a substitute for measuring your own setup, because the total depends on the model, the quantisation and the context you allow. Run systemctl show llama-server -p MemoryCurrent while a request is in flight and use the number you see.

Why does /health return 503 with "Loading model"?

The process has started but the model file is not in memory yet, so the server answers {"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}. This is normal after every restart, and it lasts as long as reading the file takes. It becomes a problem only when a client or a proxy treats that first 503 as a hard failure. Poll /health until it returns {"status": "ok" }.

Can I expose llama-server directly to the internet?

Do not bind it to 0.0.0.0 and open the port. It has no accounts, no rate limiting and no request log worth auditing, and the only built-in check is --api-key, which compares a single string. Keep the default 127.0.0.1 bind, put nginx in front with TLS, and set --api-key as well, so one mistake in the proxy config does not leave the model open to everyone.