Which AI models can you self-host?

What decides which AI models you can self-host

Which AI models you can self-host is decided by one number: the RAM on the box. The model family and the framework matter far less than whether the weights fit in memory with room left over. This post is the arithmetic for working that out. Getting a runtime installed is a separate job, covered in the guide to running Ollama on a VPS.

Two costs decide the answer. The weights are the fixed cost, set by parameter count and quantisation. The context window is the running cost, and it is the one people forget until a model that loaded yesterday refuses to load today.

The sizing arithmetic: bits per parameter

A model file is almost entirely weights. Each weight is stored at some number of bits. Quantisation means storing them at fewer bits than the precision they were trained at, which costs a little accuracy and saves a lot of memory. The size follows directly from that:

weights in GB = (parameters in billions x bits per weight) / 8

Models are released at 16 bits, which is 2 GB per billion parameters. That is why almost nobody runs release precision on a VPS. These are the quantisations you will actually meet, with their real average bits per weight:

• Q8_0 stores about 8.5 bits per weight, so roughly 1.1 GB per billion parameters.
• Q6_K stores about 6.6 bits, so roughly 0.83 GB per billion.
• Q5_K_M stores about 5.7 bits, so roughly 0.71 GB per billion.
• Q4_K_M stores about 4.8 bits, so roughly 0.6 GB per billion.

Use 0.6 GB per billion parameters as your working number. Q4_K_M is the sensible default on a memory bound box: quality loss against 8 bits is small on most tasks, and the file is nearly half the size. Below 4 bits the loss grows quickly, so a 70B squeezed to 2 bits usually answers worse than a 32B at 4 bits from the same generation. When memory is tight, drop a size class before you drop below 4 bits.

[
  {
    "label": "3B",
    "weights_gb": 1.8,
    "kv_8k_gb": 0.9,
    "kv_128k_gb": 14
  },
  {
    "label": "8B",
    "weights_gb": 4.8,
    "kv_8k_gb": 1,
    "kv_128k_gb": 16
  },
  {
    "label": "14B",
    "weights_gb": 8.4,
    "kv_8k_gb": 1.5,
    "kv_128k_gb": 24
  },
  {
    "label": "32B",
    "weights_gb": 19.2,
    "kv_8k_gb": 2,
    "kv_128k_gb": 32
  },
  {
    "label": "70B",
    "weights_gb": 42,
    "kv_8k_gb": 2.5,
    "kv_128k_gb": 40
  }
]

The weight column above is that 0.6 GB per billion rule applied. Real GGUF files land within a few percent of it, because the embedding and output layers are kept at higher precision than the rest. A 3B model at 4 bits is about 1.8 GB. An 8B is 4.8 GB. A 32B is 19.2 GB, and a 70B is 42 GB.

Why context length costs more RAM than the weights do

The KV cache (key value cache, the attention state the model keeps for every token currently in the conversation) is the second cost. It is allocated when the model loads, sized for the context length you asked for, and it grows in a straight line with that length.

bytes per token = 2 x layers x kv_heads x head_dim x bytes per element

At Ollama's default context, that 8B model spends half a gigabyte on cache. At 8192 tokens it spends 1 GB. At the 128k context its model card advertises, it spends 16 GB, which is more than three times the weights. The 70B is the opposite case: its cache at 128k is 40 GB, less than its own weights, because grouped query attention keeps the per token cost from growing anywhere near as fast as the parameter count.

Ollama's default context length is 4096 tokens on a CPU only server. When a GPU is present it picks the default from VRAM instead: 32k between 24 and 48 GiB, and 256k at 48 GiB and above. Raise it with the OLLAMA_CONTEXT_LENGTH variable on the server, then check what a running model actually got in the CONTEXT column of ollama ps. The memory arithmetic behind that one setting is worked through in the post on num_ctx and context length.

There are two ways to buy the cache back down. Ask for the context you need rather than the context the model card advertises, since most chat and coding work fits inside 8k to 32k. Or quantise the cache itself to 8 bits, which halves it, at some cost in long context recall.

A resident model holds that RAM until something unloads it

Ollama keeps a model in memory for 5 minutes after the last request, then unloads it. That default suits a laptop and is wrong for a server, where the first request after every idle gap pays the load time again.

ollama ps
ollama stop qwen3:4b

ollama ps lists what is resident, with a SIZE column showing how much memory it holds and an UNTIL column showing when it expires. To pin a model permanently, set OLLAMA_KEEP_ALIVE=-1 on the service. A value of 0 unloads it as soon as each response finishes.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Send one prompt, then run ollama ps again ten minutes later. The model is still listed, which is exactly the point: it is holding that RAM whether or not anyone is using it. A pinned model is not spare capacity. On a 16 GB VPS an 8B at 8k context holds roughly 6 GB for as long as the service runs, so size the box around the model plus your application, not around the model alone. Pinning a model in memory covers the trade against cold start latency.

What runs on a 4 GB VPS

Reserve about 1 GB for the operating system and the model server, which leaves roughly 3 GB. That is a 1B to 4B model at 4 bits, at the default 4096 token context. As of August 2026 that class includes Llama 3.2 at 3B, Qwen 3 at 1.7B and 4B, and the small Gemma and Phi releases. Treat those as size examples, not recommendations. The names turn over every few months and the arithmetic does not.

Expect roughly 6 to 14 tokens per second. Models this small do narrow work well: classification, tag extraction, short summaries, rewriting a paragraph to a house style. They are weak at multi step reasoning and at code spanning several files, and no amount of prompting fixes that.

The failure mode at this tier is swap. If the model does not fit, Linux does not refuse to load it. It pages memory out to disk instead, and because generating a single token reads every weight once, generation collapses to seconds per token. Watch free -h and the si and so columns of vmstat 1 while the model answers. Non zero swap in and swap out during generation means the model is too big for the plan.

What runs on an 8 to 16 GB VPS

This is where a self-hosted model becomes generally useful. On 8 GB you can run a 7B or 8B at 4 bits, about 4.8 GB of weights, with an 8k context. On 16 GB you can run a 13B or 14B at 4 bits, about 8.4 GB, or keep an 8B at 8 bits if you would rather spend the memory on precision than on parameter count.

Speed is the catch. An 8B on CPU generates about 3 to 7 tokens per second, and a 14B about 1.5 to 3.5. A person reads at somewhere around 5 to 10 tokens per second, so an 8B on a CPU VPS feels like watching a slow typist. That is fine for a background job and tiring for interactive chat. Measured runs of Qwen 3 at 8B and larger on a VPS show what it looks like in practice.

What runs on a 32 to 64 GB VPS

A 32B at 4 bits is about 19.2 GB, so it fits a 32 GB plan with a short context and sits comfortably on 48 GB or 64 GB. A 70B at 4 bits is about 42 GB, so it needs 64 GB before you add any cache at all.

Then read the speed honestly. A 32B on CPU runs at about 0.6 to 1.5 tokens per second, and a 70B at 0.2 to 0.5. A 500 token answer from that 70B takes about twenty minutes. These are batch tools. Feed them a queue of documents overnight and the speed does not matter. Put them behind a chat window and it matters a great deal.

Mixture of experts routing changes this arithmetic, and it is the one architecture detail worth learning. An MoE model sends each token through only a small part of its weights. A model with 30B total parameters and 3B active per token needs the memory of a 30B and generates close to the speed of a dense 3B, because each token reads only the active experts. On a 32 GB box an MoE in that shape is far more usable than a dense 30B. The rule to carry away: total parameters set the memory, active parameters set the speed.

How fast is CPU inference, honestly?

Generating one token requires reading every active weight out of memory once. Nothing avoids that, so generation speed on a CPU is set by memory bandwidth and not by core count. The ceiling is a division: usable memory bandwidth divided by the size of the weights in bytes. A small shared VPS realistically delivers 10 to 25 GB per second across its vCPUs, so a 4.8 GB model tops out near 2 to 5 tokens per second.

[
  {
    "label": "3B",
    "tokens_per_second_low": 6,
    "tokens_per_second_high": 14
  },
  {
    "label": "8B",
    "tokens_per_second_low": 3,
    "tokens_per_second_high": 7
  },
  {
    "label": "14B",
    "tokens_per_second_low": 1.5,
    "tokens_per_second_high": 3.5
  },
  {
    "label": "32B",
    "tokens_per_second_low": 0.6,
    "tokens_per_second_high": 1.5
  },
  {
    "label": "70B",
    "tokens_per_second_low": 0.2,
    "tokens_per_second_high": 0.5
  }
]

Those are ranges commonly reported on ordinary VPS hardware, not a benchmark of one machine. Your figure depends on the memory generation, the channel count on the host, and how many neighbours are competing for it. Measure your own, using any model tag you already have:

ollama run qwen3:4b --verbose "Write three sentences about disk latency."

The summary printed after the answer ends with a line reading eval rate: ... tokens/s. That is your generation speed. Ignore the first run of a session, because load duration on the same summary includes reading the weights off disk. Measuring tokens per second properly covers how to get a number worth comparing.

Two results surprise people here. Adding vCPUs stops helping quickly, because past roughly 8 cores the extra cores are waiting on memory rather than doing arithmetic. And on a shared plan the same command returns different numbers hour to hour, which is CPU steal time from a noisy neighbour rather than anything you configured wrong.

Reading your prompt is a different job from generating the answer. Prompt processing is compute bound, so it does scale with cores, and it is where a GPU pulls furthest ahead. A long document takes a CPU minutes to read and a GPU seconds.

What changes when you add a GPU

The arithmetic does not change, only the pool it applies to. VRAM is a hard limit, so work out what fits before you rent:

• 8 GB of VRAM holds a 7B or 8B at 4 bits with a short context.
• 16 GB holds a 14B at 4 bits with a real context, or an 8B at 8 bits.
• 24 GB holds a 32B at 4 bits with the context kept short.
• 48 GB and above holds a 70B at 4 bits with room for cache and concurrency.

When a model does not fit, Ollama splits it: some layers on the GPU, the rest on the CPU. ollama ps reports the split in its PROCESSOR column, as something like 78%/22% CPU/GPU. Read that as a warning rather than a feature. The CPU half sets the pace, because every token still waits on those layers, so a model with a quarter of its layers on the CPU runs much closer to CPU speed than to GPU speed. If you see a split you did not intend, lower the context length first. The cache is usually what pushed it over.

Concurrency is the other reason to size up. The weights are shared between simultaneous requests, but every active request needs its own KV cache, so ten concurrent users of an 8B at 8k context need ten times 1 GB of cache on top of the weights. Serving concurrent users from one self-hosted model works through where that ceiling lands.

Whether the GPU is worth renting is an arithmetic question too, and it comes down to how many tokens per month you really generate. The break even between a GPU VPS and API tokens has those numbers.

What you cannot self-host

There are two different walls here, and it helps to know which one you are hitting.

The first is closed weights. Frontier commercial models are not distributed, so there is no file to download and no amount of RAM changes that. You can self-host everything around them: the interface, the retrieval layer, the agent loop, the logs. The model itself stays a remote API. Whether you can self-host Claude goes through that in full.

The second is open weights that are simply too large. The biggest open releases are mixture of experts designs in the hundreds of billions of total parameters. The same rule applies to them: a 400B total parameter model at 4 bits needs about 240 GB for weights alone, before any cache. That is specialist hardware, and renting it by the month costs far more than most people spend on API tokens in a year. What it takes to self-host a Kimi class model walks through the real requirement.

The honest line between the two: self-host when the load is steady and the data should not leave your server. Buy tokens when the load is bursty, or when frontier answer quality is the thing you actually need.

Check what you have before you choose

free -h
nproc
lscpu | grep 'Model name'

Plan against the available column of free -h, not the total column, because total includes memory the system is already using. Subtract about 1 GB for the operating system and the model server. Divide what is left by 0.6 to get the largest parameter count in billions you can hold at 4 bits. Then subtract the KV cache for the context you actually want. What remains is your answer, and unlike a list of model names it does not go stale.

FAQ

How much RAM do I need to run an 8B model?

About 4.8 GB for the weights at 4 bit quantisation, plus the KV cache for your context length, plus roughly 1 GB for the operating system and the model server. At an 8192 token context the cache adds about 1 GB, so an 8 GB plan works and a 4 GB plan does not. If you want the full 128k context the model card advertises, the cache alone is 16 GB and you are looking at a 32 GB plan.

Why is my model slow even though the VPS has plenty of vCPUs?

Because generation is limited by memory bandwidth, not by cores. Every token requires pulling the whole active weight set out of RAM, so once a few cores saturate the memory channels the rest just wait. The other common cause is swap. If vmstat 1 shows non zero si and so while the model is answering, the weights do not fit in RAM and part of every token is being served from disk, which costs far more than it looks like it should.

Does a longer context window really need more memory?

Yes, and the growth is linear in tokens. A typical 8B model spends about 128 KiB of KV cache per token, so 8192 tokens costs 1 GB and 131072 tokens costs 16 GB. The cache is allocated when the model loads rather than when the conversation grows, so asking for a 128k context reserves that memory immediately, even if every prompt you send is 200 tokens long.

Should I run a large model at 2 bits or a smaller one at 4 bits?

Take the smaller model at 4 bits. Quality falls slowly from 8 bits down to 4 and quickly below 4, so a 70B squeezed to 2 bits usually gives worse answers than a 32B at 4 bits from the same model generation. Heavy quantisation shows up as repetition and dropped instructions rather than as an error message, which makes it easy to blame on your prompt. Treat 4 bits as the floor and change the parameter count instead.

Can I self-host a model as capable as the big commercial ones?

Not on an ordinary VPS. The strongest open weight models run to hundreds of billions of parameters, which at 4 bits means over 200 GB of RAM before any KV cache, and the strongest commercial models are not distributed at all. What ordinary hardware does well is run a good 8B to 32B model for one specific job, where a narrow and well prompted small model often matches a general one. If you need frontier quality, price the API against the hardware before you buy either.