How big an LLM fits in your server RAM

How much RAM does an LLM need?

How big an LLM fits in your server RAM comes down to four numbers added together: the weights, the KV cache, the runtime's working buffers, and the headroom the operating system needs. The weights dominate, and they are pure arithmetic, parameter count times bits per weight, divided by eight. If the total sits under your physical RAM, the model loads.

Loading is the easy half. On a server with no GPU the rate of generation is set by memory bandwidth, so a model that fits can still be too slow for the job you had in mind. Sizing you can do on paper. Speed you have to measure on your own box.

Every figure below is arithmetic on published parameter counts and published quantisation levels, checked on 2 September 2026. None of it is a benchmark result.

The arithmetic: parameters times bits per weight

One formula does most of the work.

weight bytes = parameters x bits per weight / 8

A model with 8 billion parameters at 16 bits per weight is 8,000,000,000 x 16 / 8 = 16,000,000,000 bytes. That is 16 GB in the units a download page uses and 14.9 GiB in the units free -h uses. A gibibyte (GiB) is 2^30 bytes and a gigabyte (GB) is 10^9 bytes, so the same file looks about 7 percent smaller when it is reported in GiB. This guide uses GiB throughout, because that is what your server reports.

That gives a shortcut worth memorising: at 16 bits per weight, a model costs about 1.86 GiB per billion parameters. Halve the bits and you halve the gibibytes.

[
  {
    "label": "8B dense",
    "fp16_gib": 14.9,
    "q8_gib": 7.9,
    "q4km_gib": 4.5
  },
  {
    "label": "14B dense",
    "fp16_gib": 26.1,
    "q8_gib": 13.9,
    "q4km_gib": 7.9
  },
  {
    "label": "32B dense",
    "fp16_gib": 59.6,
    "q8_gib": 31.7,
    "q4km_gib": 18.0
  },
  {
    "label": "70B dense",
    "fp16_gib": 130.4,
    "q8_gib": 69.3,
    "q4km_gib": 39.4
  }
]

Those rows are computed, not measured. They are nominal parameter counts multiplied out at 16, 8.5 and 4.83 bits per weight. A 70B model at full precision needs 130.4 GiB for weights alone, which is why nobody runs one that way on a general purpose VPS. The same 70B at Q4_K_M needs 39.4 GiB, and that is a machine you can actually rent.

Quantisation: what the bits per weight really are

Q4_K_M does not store four bits per weight. GGUF k-quants pack weights into blocks, and every block carries its own scale and minimum value. Some tensors, the embeddings and the attention output among them, are kept at higher precision because shrinking those costs more quality than it saves memory. The published average for Q4_K_M is about 4.83 bits per weight, and Q8_0 is about 8.5. Those are the values behind the chart above.

You do not have to trust an average. A GGUF file on disk is exactly the tensor data you are about to load, so its size is the honest figure. After an ollama pull, look at both:

ollama show qwen3:8b
ls -lh /usr/share/ollama/.ollama/models/blobs | sort -k5 -h | tail -3

ollama show prints the architecture, the parameter count and the quantisation of the model you asked for. The blob listing gives the real file size, which is the number your RAM has to cover. The path above is the one the systemd service uses; if you run ollama serve as your own user the blobs live in ~/.ollama/models/blobs instead. An empty listing means you are looking at the wrong one.

Which level to pick is a quality question rather than a memory question, and the difference Q4, Q8 and FP16 make to the output is worth reading before you settle the memory side.

The KV cache grows with context

The KV cache (key/value cache) holds the attention keys and values for every token already in the sequence, so the model does not recompute them for each new token. It is allocated per sequence and it grows in a straight line with the context length.

kv bytes per token = 2 x layers x kv_heads x head_dim x bytes per element

The leading 2 counts keys and values separately. Take Qwen3-8B, whose published config reports 36 layers, 8 key/value heads and a head dimension of 128. With an f16 cache at 2 bytes per element:

2 x 36 x 8 x 128 x 2 = 147,456 bytes per token = 144 KiB per token

[
  {
    "label": "4k context",
    "kv_cache_gib": 0.56
  },
  {
    "label": "8k context",
    "kv_cache_gib": 1.13
  },
  {
    "label": "16k context",
    "kv_cache_gib": 2.25
  },
  {
    "label": "32k context",
    "kv_cache_gib": 4.5
  }
]

At a 4,096 token window the cache is 0.56 GiB, which is noise next to the weights. At 32,768 tokens it is 4.5 GiB, and on a 16 GiB server that is more than a quarter of the machine spent on conversation history.

Two multipliers make this worse fast.

Concurrency. Every request in flight needs its own cache. Ollama's OLLAMA_NUM_PARALLEL defaults to 1, and raising it to 4 multiplies the KV cache by four, because four sequences each hold their own keys and values. This is the wall you hit long before the CPU runs out of work, and it drives how many people one self-hosted model can serve at once.

Attention shape. Qwen3-8B uses grouped query attention (GQA) with 8 key/value heads against 32 query heads. An older model of the same depth using full multi head attention has 32 key/value heads, so its cache costs four times as much per token. Two models with the same parameter count can differ by that factor, so read the config rather than assuming.

You have two levers. Lower the context, which is the cheapest change available and is covered in setting num_ctx without truncating your own prompts. Or quantise the cache itself:

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

OLLAMA_KV_CACHE_TYPE accepts f16, q8_0 and q4_0, and it needs flash attention switched on to take effect. Moving from f16 to q8_0 roughly halves the cache, because each stored element drops from two bytes to about one.

Runtime overhead and headroom for the operating system

Past weights and cache, the runtime allocates compute buffers for the batch it is working on, and those scale with context and batch size. Budget about 1 GiB for a single user on a small model, then check it instead of trusting the estimate.

The operating system needs headroom too. A headless Ubuntu server idles at a few hundred megabytes, but the page cache matters more than the idle footprint. The model file is read through it, so loading a 9 GiB model on a 16 GiB box evicts nearly everything the machine had cached. Leave 2 GiB free and you avoid the worst of that.

Ollama maps the model file rather than copying it, which makes free -h awkward to read. The resident set can look smaller than the model while the kernel counts those pages as file backed. The reliable check is the runtime's own view:

ollama ps

That prints each loaded model with a SIZE column and a PROCESSOR column. SIZE is what the runtime reserved in total, including the KV cache. On a server with no GPU, PROCESSOR reads 100% CPU. If SIZE is close to your total RAM, you have no headroom left and the next long prompt is the one that breaks it.

Worked example: what fits in 16 GB of RAM

Start with a 16 GiB server, no GPU, nothing else running on it.

1. Reserve 2 GiB for the operating system and the page cache. 14 GiB left.
2. Reserve 1 GiB for runtime buffers. 13 GiB left.
3. Choose an 8k context on a GQA model, which costs 1.13 GiB from the chart above. Just under 12 GiB left.
4. Divide those 12 GiB by the 0.56 GiB that one billion parameters costs at Q4_K_M. That is room for roughly 21 billion parameters of weights.

So a 14B model at Q4_K_M, at 7.9 GiB, fits with several gigabytes to spare, and the spare capacity can go back into context. A 32B at the same quantisation wants 18.0 GiB and does not fit at all, however short you make the context. That same 32B at Q8_0 wants 31.7 GiB, which is twice the machine.

Run those four steps against your own figure from free -h and you have your answer. The shortlist of open weight models worth self-hosting is where to pick a candidate, and installing Ollama on a VPS covers getting the runtime in place first.

Mixture of experts: big in memory, small per token

A dense model multiplies every weight for every token. A mixture of experts (MoE) model replaces the feed forward block in each layer with many expert networks plus a router that picks a few of them per token. Qwen3-30B-A3B has 128 experts per layer and activates 8 of them for each token.

[
  {
    "label": "Qwen3-8B dense",
    "total_params_b": 8.2,
    "active_params_b": 8.2
  },
  {
    "label": "gpt-oss-20b MoE",
    "total_params_b": 21.0,
    "active_params_b": 3.6
  },
  {
    "label": "Qwen3-30B-A3B MoE",
    "total_params_b": 30.5,
    "active_params_b": 3.3
  },
  {
    "label": "gpt-oss-120b MoE",
    "total_params_b": 117.0,
    "active_params_b": 5.1
  }
]

Those are the counts published on each model card, read on 2 September 2026. The gap between the two series is the whole point. Qwen3-30B-A3B keeps 30.5 billion parameters resident and multiplies only 3.3 billion of them per token. gpt-oss-120b keeps 117.0 billion resident and activates 5.1 billion.

Size an MoE model from the total, because the router can choose any expert at any token, so all of them have to be in memory. Speed follows the active count, because that is what gets read out of memory for each token. An MoE model therefore buys speed with RAM, which is a good trade on a CPU server when the RAM is there and a bad one when it is not.

The failure mode here is quiet. If the weights do not fit, nothing refuses to run. The kernel faults expert pages in from storage as the router asks for them, and because the router changes its mind at every token, the working set never settles. The model keeps answering, and each token now waits on the disk instead of on memory. Watch vmstat 1 while it generates and you will see exactly that.

Fitting is not the same as usable

Generating one token means reading every active weight out of memory once. That makes a CPU-only setup bandwidth bound: the ceiling on tokens per second is your memory bandwidth divided by the active weight bytes per token. Cores past the point where the memory controller saturates buy nothing, which is why adding cores to a CPU-only box stops helping well before you run out of cores.

That is why no tokens per second figure belongs in a sizing guide. Memory bandwidth varies by CPU generation and by how many memory channels the host has populated, and on a VPS it also moves with what the other tenants are doing at that moment. sudo dmidecode -t memory on a virtual machine usually reports the hypervisor's synthetic values rather than the physical modules, so it will not settle the question either. The only number that means anything is the one your own box produces under your own prompt, and measuring tokens per second properly on your own machine is a short job with a clear result.

Prompt processing behaves differently from generation. Reading a long prompt is compute bound and runs across many tokens in parallel. Generating the reply is bandwidth bound and strictly serial. So a large context charges you twice: gigabytes of KV cache, and a wait before the first token appears.

What it looks like when the model does not fit

The OOM killer takes the process. Check with journalctl -k | grep -i 'out of memory' and you will find a line like this:

Out of memory: Killed process 2841 (ollama) total-vm:14839216kB, anon-rss:12106884kB

The client usually sees the connection drop with no error body, so the kernel log is the only place the reason exists.

It loads, then swaps. Run vmstat 1 5 during generation. The si and so columns are pages swapped in and out per second. Any sustained non-zero value there means weights are being read from the swap device for every token, and no amount of tuning fixes that. Turning swap off does not fix it either. It converts a slow model into the OOM kill above, which is at least honest.

The context is quietly smaller than you asked for. Ollama's OLLAMA_CONTEXT_LENGTH defaults to 4096 tokens. Send a longer conversation without raising it and the request is served against that window, so the older turns fall out and the model answers as though they never happened. Nothing errors. The only symptom is a model that seems to forget.

First load takes minutes, then it is fine. That is the file being read from disk into the page cache, not a fit problem. OLLAMA_KEEP_ALIVE controls how long the model stays resident afterwards, and a negative value keeps it loaded indefinitely.

Shrink the model, shrink the context, or leave the CPU

Work down this list in order, because the early items cost you least.

1. Cut the context. Going from a 32k window to 8k gives back most of 4.5 GiB on an 8B model, and it costs nothing if your prompts were never that long.
2. Quantise the KV cache to q8_0. That roughly halves whatever is left of it.
3. Drop one quantisation level on the weights, Q8_0 down to Q4_K_M. This is the largest single saving available and the one with a real quality cost.
4. Choose a smaller model, or an MoE model with a small active count if you have the RAM for its total.
5. Stop buying RAM and change platform.

Point five deserves the plain version. RAM capacity decides whether the model loads. Memory bandwidth decides how fast it answers. If you have measured your box and the rate is too slow for the work, more RAM will not move it, because capacity was never the constraint. That is the moment to price a GPU instance against a hosted endpoint. Work out your tokens per month, multiply by the provider's published rate, and compare that against the monthly cost of a GPU plan large enough to hold the model. At low volumes the hosted endpoint usually wins on cost alone, and what you give up by sending tokens to somebody else's server covers the part of that decision that is not about money.

A CPU server keeps its place for work where latency does not matter, such as summarisation that runs overnight or a local assistant serving one person. For that kind of workload the arithmetic above is most of the engineering.

FAQ

How much RAM do I need to run an 8B model on a CPU?

Budget 4.5 GiB of weights for an 8 billion parameter model at Q4_K_M, about 1.1 GiB of KV cache at an 8k context, roughly 1 GiB of runtime buffers, and 2 GiB for the operating system. That lands near 8.6 GiB, so an 8 GiB server is too tight and a 16 GiB server is comfortable. At Q8_0 the weights alone are 7.9 GiB, which pushes the same setup to about 12 GiB and leaves no room to grow the context.

Does adding RAM make a CPU-only LLM faster?

Once the model fits without swapping, no. Generation speed is set by memory bandwidth and by how many bytes the model reads per token, and unused capacity changes neither of those. The one case where extra RAM looks like a speedup is a machine that was swapping before, because reading weights from a disk is far slower than reading them from memory. Measure the rate before and after any change rather than assuming which case you are in.

Why does the model load fine and then run out of memory during a long chat?

Because the KV cache grows with the conversation while the weights stay fixed. The same model holds 0.56 GiB of cache at a 4,096 token window and 4.5 GiB at 32,768 tokens. Some runtimes allocate the full window up front, so the failure arrives at load time. Others grow it as the sequence grows, so the failure arrives mid conversation. Lower num_ctx, or quantise the cache with OLLAMA_KV_CACHE_TYPE=q8_0 and OLLAMA_FLASH_ATTENTION=1.

Can I run a 70B model on a CPU server with enough RAM?

It will load. A 70B at Q4_K_M needs 39.4 GiB of weights, so a 64 GiB server is short and a 96 GiB server has room once you add cache and headroom. Whether it is usable is a separate question, because a dense 70B reads all of those gigabytes for every single token and that rate is capped by memory bandwidth. Measure it on your own box before you commit to the plan. An MoE model of similar total size is often the better answer, since it needs the same RAM but reads far fewer bytes per token, and choosing between Qwen 3 sizes on a VPS walks through that trade.