Run Ollama on a VPS: self-host an LLM

What you are building

A single open-weight language model running on a server you own, answered over an HTTP API and, if you want, a chat page in your browser. Ollama is the piece that downloads the model, loads it into memory, and serves requests on http://127.0.0.1:11434. The install is one command. Everything hard about this lives elsewhere: picking a model your VPS can actually hold in RAM, and not accidentally publishing an unauthenticated inference server to the whole internet.

Two honest warnings first. A CPU-only VPS runs small models slowly, and there is no built-in authentication on the API at all. Both are covered in detail below, because both are where people get hurt.

The sizing reality check, in plain numbers

A model's memory footprint is roughly its file size, plus about a gigabyte of runtime overhead, plus some more for the context window. Ollama's default models are 4-bit quantized (labelled Q4), which costs about half a gigabyte of RAM for every billion parameters. So the arithmetic is simple, and it decides everything.

A 3B model such as llama3.2:3b is a ~2 GB download and wants around 4 GB of free RAM to run. A 7B or 8B model such as mistral:7b or llama3.1:8b is ~5 GB on disk and wants about 8 GB of RAM, 16 GB to be comfortable. A 13B or 14B model wants roughly 16 GB. Anything in the 30B-to-70B range needs a large-RAM box or, realistically, a GPU — on a CPU VPS it either will not fit or will answer so slowly it is useless.

Now speed, because this is the part people underestimate. CPU inference is bound by memory bandwidth, not clock speed, and a shared vCPU VPS has modest bandwidth. Expect single-digit to low-double-digit tokens per second: a 7-8B Q4 model might manage 4 to 10 tokens per second, a 3B model 10 to 25. A GPU is roughly an order of magnitude faster. These are rough figures on purpose — the honest move is to measure your own box, which the run step below shows how to do. Trust your eval rate, not a number in any article, this one included.

The practical conclusion: small quantized models on CPU are genuinely useful for drafting, summarizing, and classification if you can accept the pace. For anything larger or faster, budget for a GPU instance.

Install Ollama

There are two clean ways. The official script is simplest on a bare VPS:

curl -fsSL https://ollama.com/install.sh | sh

This creates a system user named ollama, installs the binary to /usr/local/bin/ollama, and registers a systemd service called ollama.service that starts on boot and binds 127.0.0.1:11434. Confirm it is up:

systemctl status ollama
ollama --version

If you already run Docker, use the container instead:

docker run -d --name ollama \
  -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama \
  --restart always \
  ollama/ollama

Note the 127.0.0.1: prefix on the port mapping. That binds the port to localhost only. Writing -p 11434:11434 instead publishes it on every interface, which is the mistake the security section warns about. Pick one install method; do not run the script and the container at the same time, or two processes fight over the port.

Pull and run your first model

ollama pull llama3.2:3b
ollama run llama3.2:3b

pull downloads the model layers to disk (about 2 GB for this one). run loads them into memory and drops you at a >>> prompt. Type a question. The first token may take several seconds while the weights load from disk into RAM, then the answer streams. Type /bye to leave the chat; Ollama keeps running in the background.

See what is loaded and how it fits:

ollama ps

The PROCESSOR column tells the truth. 100% CPU means no GPU is involved, and that is where the slowness comes from. Measure real speed with the verbose flag:

ollama run --verbose llama3.2:3b "Write two sentences about Linux."

The eval rate line printed at the end is your tokens per second on this hardware. That is the number to plan around.

Where models live, and how much disk to buy

Installed by the script and run as the service, models live in the ollama user's home:

sudo du -sh /usr/share/ollama/.ollama/models

Run interactively as your own user, they sit in ~/.ollama/models. In the container they live in the ollama named volume. This matters because quantized weights add up fast: a 3B is ~2 GB, a 7-8B is ~5 GB, a 14B is ~9 GB. Pull four models to compare them and you have spent 20 GB without noticing. Size the disk for the models you mean to keep, and delete the rest with ollama rm <model>.

Run it as a service you control

The install script already registered ollama.service, so it restarts on boot without further work. The setting worth changing is how long a model stays resident, and, on some setups, the bind address — both go in a systemd drop-in so an Ollama upgrade does not overwrite them:

sudo systemctl edit ollama.service

Add this under the [Service] header the editor shows you:

[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"

OLLAMA_KEEP_ALIVE is how long a model stays in memory after the last request (default 5 minutes). Raise it on a box you query all day to avoid reloading the weights each time; set it to 0 on a tight box to free RAM the moment a request finishes. systemctl edit reloads the unit files for you, so restart to apply the change:

sudo systemctl restart ollama

The security point that matters most

By default Ollama binds 127.0.0.1:11434, so only processes on the VPS itself can reach it. That default is correct. Keep it.

The API has no authentication. None. There is no API key, no login, no rate limit, no allow-list. Anyone who can reach port 11434 can run any model you have pulled, pull new ones, delete them, and pin your CPU or GPU at full load indefinitely. Scanners such as Shodan index open Ollama instances by the thousand, and an exposed one is found and abused within hours.

So here is the single mistake never to make: do not set OLLAMA_HOST=0.0.0.0 and open 11434 in your firewall. That publishes an unauthenticated inference server to the entire internet. No amount of configuration makes raw 11434-on-0.0.0.0 safe, because there is nothing in Ollama to configure — the authentication simply does not exist.

There are three safe ways to reach the model from somewhere other than the box:

• Keep it local. If the only caller is another program on the same VPS — a cron script, a bot, an MCP server bridging your tools to the model — leave the bind at 127.0.0.1 and have that program call http://127.0.0.1:11434. Nothing is exposed and nothing else is needed.
• Reach it over a private tunnel. Put the VPS on a WireGuard VPN you host yourself, set OLLAMA_HOST to the tunnel address (for example 10.8.0.1, not 0.0.0.0), and only VPN peers can connect. The public internet still sees nothing on 11434.
• Put an authenticating reverse proxy in front. Terminate TLS and require a password or token at nginx, Traefik, or Caddy, then proxy to 127.0.0.1:11434. Ollama keeps its localhost bind; the proxy is the only thing listening on the public port. This is the same shape as putting a Let's Encrypt certificate on nginx in front of any local service.

The reverse-proxy option is exactly what the chat UI gives you next, with a real login attached.

Add a chat UI with Open WebUI, behind TLS

Open WebUI is a self-hosted chat interface. Run it in Docker and point it at the local Ollama:

docker run -d \
  --name open-webui \
  --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

The --network=host flag is the important detail on a Linux VPS. It puts the container in the host's network namespace, so 127.0.0.1 inside the container is the host's own loopback and the container reaches Ollama at 127.0.0.1:11434 without Ollama listening on any other interface. The bridge-network recipe you will see elsewhere — --add-host=host.docker.internal:host-gateway with OLLAMA_BASE_URL=http://host.docker.internal:11434 — does not work here: that name resolves to the Docker bridge gateway, and a service bound to 127.0.0.1 on the host is not reachable across the bridge, so Open WebUI just sits there reporting it cannot connect to Ollama.

The trade-off of host networking is that Open WebUI now listens on the host's port 8080 on every interface; any -p mapping is discarded, and Docker prints a warning saying so. So close 8080 at both the host and provider firewall and let the TLS reverse proxy be the only public door. On the very first visit Open WebUI asks you to create an admin account — that account is your authentication layer, so choose a strong password.

To open the chat from your laptop over HTTPS, put a TLS reverse proxy in front of 127.0.0.1:8080. If you already route several Docker apps on the box, Traefik with automatic TLS across many apps is the cleanest fit: one label block issues the certificate and routes chat.example.com to Open WebUI. The rule from the security section still holds — the proxy owns the public port and the login, while Ollama stays on localhost and Open WebUI's own 8080 stays firewalled.

Use the OpenAI-compatible endpoint from your code

Ollama speaks a subset of the OpenAI chat API at /v1, so most OpenAI client libraries work after changing two things: the base URL and a throwaway key.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Name three Linux distributions."}],
)
print(resp.choices[0].message.content)

The api_key is required by the client library but ignored by Ollama, so any string works. model must be a name you have already pulled; an unknown name returns model "x" not found, try pulling it first. A plain curl call is the same idea:

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:3b","messages":[{"role":"user","content":"Hello"}]}'

This is also how you wire the model into agent and editor tooling. If you already develop on the box, a local model can back scripts and plugins alongside Claude Code running on the VPS inside tmux, keeping cheap, private drafting work off a paid API while the heavy reasoning stays with a hosted model.

Failure modes, with the exact strings you will see

The process is "Killed" mid-generation. You start a large model and the terminal prints Killed, or the server log shows llama runner process has terminated: signal: killed. The Linux OOM killer stopped it because the model needed more RAM than the box has. Confirm the cause with sudo dmesg | grep -i oom, where you will see a line like Out of memory: Killed process ... (ollama). The fix is a smaller or more heavily quantized model — llama3.2:3b instead of a 13B — or adding swap so a load that only just overflows physical RAM survives slowly instead of dying. Swap turns an instant crash into a slow answer; it does not make a 70B model practical on 4 GB.

"Error: model requires more system memory". Ollama refuses to start the model and prints Error: model requires more system memory (X GiB) than is available (Y GiB). This is the polite version of the crash above: Ollama did the arithmetic and stopped rather than letting the OOM killer do it. It even hands you the two numbers. Choose a model whose requirement is under your free RAM (check with free -h), shrink the context length, or move to a larger VPS. No flag makes the model fit — the memory is real.

The first token takes forever, then it is fine. A cold model prints nothing for five to thirty seconds, then streams normally. That pause is the weights loading from disk into RAM for the first time, and slow storage makes it worse. Once loaded, the model stays resident for the length of OLLAMA_KEEP_ALIVE, so the second prompt answers instantly. Raise that value if the gaps annoy you, and use ollama ps to see whether a model is currently loaded.

Everything is simply slow. Ten tokens a second or fewer, with no error at all. That is CPU inference doing exactly what CPU inference does. ollama ps shows 100% CPU, meaning there is no GPU. This is not a bug and no setting fixes it, because the limit is memory bandwidth, not a misconfiguration. Use a smaller model, accept the speed, or move to a GPU instance — and measure your real rate with --verbose before deciding anything is broken.

Connection refused from another machine. From your laptop you get curl: (7) Failed to connect to <ip> port 11434: Connection refused. This is working as designed: Ollama binds localhost only. Do not "fix" it by binding 0.0.0.0, which is precisely the exposure mistake above. Reach the model over the VPN or through the authenticating proxy instead.

You exposed 11434 to the internet. If you did set OLLAMA_HOST=0.0.0.0, opened the firewall, and now see model pulls you never started or the CPU pinned at 100% by unknown clients, you have been found and used. This is the headline mistake, not an edge case. Rebind to 127.0.0.1 or the VPN address, close 11434 at the firewall, and put authentication in front. Assume anything reachable at that address while it was open was queried by strangers.

Backups and upgrades

There is little state to lose. The models are re-downloadable, so the only things worth backing up are Open WebUI's data volume — accounts, chat history, settings — and any systemd drop-in you wrote. Back up the volume with a throwaway container:

docker run --rm -v open-webui:/data -v "$PWD":/backup alpine \
  tar czf /backup/open-webui.tgz -C /data .

Upgrade Ollama by re-running the install script; upgrade Open WebUI with docker pull ghcr.io/open-webui/open-webui:main followed by recreating the container. Pin nothing long-term: both model quality and the runtime move fast, so read the release notes and re-benchmark on your own box rather than trusting last quarter's numbers.

FAQ

Can I really run an LLM on a CPU-only VPS?

Yes, within limits. Small quantized models in the 3B to 8B range run on CPU and are genuinely useful for drafting, summarizing, and classification — just slowly, at single-digit to low-double-digit tokens per second on a shared vCPU. Anything from 13B upward is painfully slow or will not fit in RAM at all. For real speed or larger models you need a GPU instance.

How much RAM does each model need?

A rough rule for the default 4-bit quantized models: about 0.5 GB of RAM per billion parameters for the weights, plus roughly 1 GB of overhead and a little more for context. So a 3B model wants about 4 GB free, a 7-8B model about 8 GB, and a 14B model about 16 GB. Check your headroom with free -h and leave room for the operating system and anything else on the box.

Is the Ollama API authenticated?

No. Ollama has no built-in authentication, API key, or rate limit — anyone who can reach port 11434 has full control of it. That is exactly why it binds 127.0.0.1 by default and why you must never expose 11434 on 0.0.0.0 to the internet. Reach it locally, over a private VPN, or through a reverse proxy that adds a login.

How do I add a web chat interface?

Run Open WebUI in Docker with --network=host so it shares the host's loopback and reaches native Ollama at http://127.0.0.1:11434, then put a TLS reverse proxy in front of its port 8080 for access from your laptop. Keep 8080 closed at the firewall so the proxy is the only public door. Open WebUI's own admin account supplies the login, and you set its password on first launch.

How do I call it from my own application?

Use the OpenAI-compatible endpoint at http://127.0.0.1:11434/v1. Point any OpenAI SDK at that base URL, pass any string as the API key since it is ignored, and set model to a name you have pulled. Existing OpenAI code usually runs unchanged apart from the base URL and key.