Ollama quantization: Q4 vs Q8 vs fp16

What Ollama quantization changes

Ollama quantization stores every weight in a model at fewer bits than the file it was trained in. A tag ending in q4_K_M keeps about four bits per weight where fp16 keeps sixteen, so the download is roughly a quarter of the size and the machine reads a quarter as many bytes to produce each token. The weights are rounded onto a coarse grid, not thrown away, and at four bits most models answer close to the way they did at full precision.

That is the whole trade: a much smaller memory footprint and more tokens per second, paid for with a small loss of accuracy. What follows is how to predict both sides of it for a specific model on a specific box, before you spend twenty minutes pulling a file that will not fit.

If Ollama is not running yet, start with installing Ollama on a VPS. This page assumes ollama ls already works.

How to read an Ollama quantization tag like q4_K_M

Local models ship as GGUF files, the format llama.cpp uses to store weights on disk. Ollama is built on llama.cpp, so Ollama tags carry llama.cpp's quantization names unchanged.

The number is the target width. q4 means most weight tensors are packed at four bits each. q8 means eight. fp16 is not quantized at all: it is the model at sixteen bit floating point, the precision most models are published in.

K marks a K-quant. Weights are grouped into small blocks, and each block stores its own scale next to the packed values. A block whose weights all sit near 0.01 gets a fine scale. A block holding one large outlier gets a coarse one. Those per block scales are what keeps a four bit file usable, and they are also the reason a four bit file is never exactly four bits per weight.

The last letter is the mixture. S, M and L decide how many tensors are promoted above the target width. In q4_K_M the tensors that hurt most when rounded are stored wider, while the bulk stays at four bits. That is why q4_K_M produces better output than the older q4_0 at almost the same file size.

Ask Ollama what it has on disk rather than guessing from the name you typed:

ollama pull qwen3:8b-q4_K_M
ollama show qwen3:8b-q4_K_M

ollama show prints architecture, parameters, quantization, context length and embedding length. The quantization line is the ground truth for a model you pulled months ago and no longer remember choosing.

Bits per weight is where the file size comes from

Every size estimate starts from one number: how many bits the format spends per weight, averaged across the whole file. llama.cpp publishes measured figures for Llama 3.1 8B in its quantize documentation, and they transfer well to any dense model of a similar shape.

[
  {
    "label": "F16",
    "bits_per_weight": 16,
    "file_gib": 14.96
  },
  {
    "label": "Q8_0",
    "bits_per_weight": 8.5,
    "file_gib": 7.95
  },
  {
    "label": "Q6_K",
    "bits_per_weight": 6.56,
    "file_gib": 6.14
  },
  {
    "label": "Q5_K_M",
    "bits_per_weight": 5.7,
    "file_gib": 5.33
  },
  {
    "label": "Q4_K_M",
    "bits_per_weight": 4.89,
    "file_gib": 4.58
  },
  {
    "label": "Q3_K_M",
    "bits_per_weight": 3.99,
    "file_gib": 3.74
  }
]

The surprise in that table is the second column. Q4_K_M is not four bits per weight. It measures 4.89 bits, because the block scales and the promoted tensors both cost real space. Q8_0 measures 8.5 bits rather than eight, for the same reason. Use the measured number and the arithmetic lands within a few percent of the real file:

weight bytes = parameter count x bits per weight / 8
8.03e9 params x 4.89 bits / 8 = 4.91e9 bytes = 4.57 GiB

That is the 4.58 GiB Q4_K_M file, recovered from two numbers. It is also, near enough, the memory the weights occupy once loaded. Ollama does not unpack anything on load: the quantized weights sit in memory in the same packed form and each block is converted as it is used.

What Ollama actually ships for each model size

The library publishes a q4_K_M, a q8_0 and an fp16 tag for most families. These are the Qwen3 sizes as of August 2026, read from the tag list on the model page.

[
  {
    "label": "Qwen3 4B",
    "q4_K_M_gb": 2.6,
    "q8_0_gb": 4.4,
    "fp16_gb": 8.1
  },
  {
    "label": "Qwen3 8B",
    "q4_K_M_gb": 5.2,
    "q8_0_gb": 8.9,
    "fp16_gb": 16
  },
  {
    "label": "Qwen3 14B",
    "q4_K_M_gb": 9.3,
    "q8_0_gb": 16,
    "fp16_gb": 30
  },
  {
    "label": "Qwen3 32B",
    "q4_K_M_gb": 20,
    "q8_0_gb": 35,
    "fp16_gb": 66
  }
]

The default tag matters here. ollama pull qwen3:8b downloads exactly the same 5.2 GB as ollama pull qwen3:8b-q4_K_M, because the unsuffixed tag is the q4_K_M build. Q4_K_M is not a compromise the library grudgingly offers. It is the default upstream chose, so matching it is the sensible first move for any model you have not tested yourself. The same reasoning drives the tag choices in running Qwen 3 on a VPS.

The ratios hold across every row. Moving from q4_K_M to q8_0 costs about seventy percent more rather than exactly double, because the embedding and output tensors do not scale the same way as the rest. fp16 is roughly three times q4_K_M. A 32B model at q4_K_M is 20 GB of weights, which is already past what a 16 GB box can hold with any context window at all. For a wider view of what fits which machine, see which models you can self-host.

Why the KV cache is a second, context dependent cost

Weights are the fixed cost. The KV cache (key and value cache) is the variable one. Every token in the context window keeps its key and value vectors for every layer, so the cache grows in a straight line with the window you allow. It is allocated for the whole window when the model loads, not as the conversation fills up, which is why a long window costs you memory even on a one word prompt.

KV bytes per token = 2 (key and value) x layers x kv_heads x head_dim x bytes_per_element
Qwen3 8B at f16:  2 x 36 x 8 x 128 x 2 = 147456 bytes = 144 KiB per token

Those model numbers come from the model's own configuration: 36 layers, 8 key/value heads, and a head dimension of 128. ollama show gives you the architecture and parameter count, and the model's config.json on Hugging Face gives you the rest. Multiply per token cost by window size and the cache stops being a rounding error.

[
  {
    "label": "4k",
    "kv_cache_gb": 0.6,
    "floor_ram_gb": 5.8
  },
  {
    "label": "8k",
    "kv_cache_gb": 1.21,
    "floor_ram_gb": 6.4
  },
  {
    "label": "16k",
    "kv_cache_gb": 2.42,
    "floor_ram_gb": 7.6
  },
  {
    "label": "32k",
    "kv_cache_gb": 4.83,
    "floor_ram_gb": 10
  }
]

At Ollama's default window of 4096 tokens the cache adds 0.6 GB on top of the weights. Raise the window to 32k and the cache alone reaches 4.83 GB, which is almost as much memory as the quantized weights, and the floor for the whole model becomes 10 GB. Call it a floor because compute buffers and the operating system sit on top of it. Read the real figure from the SIZE column of ollama ps after the model loads.

The window is set on the server, not per request, when you run Ollama as a service:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

For a systemd install, put it in a drop-in instead:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Restart with sudo systemctl restart ollama, then check the CONTEXT column of ollama ps to confirm the window the running model was actually loaded with. OLLAMA_KV_CACHE_TYPE quantizes the cache itself: f16 is the default, q8_0 uses about half the memory of f16, and q4_0 about a quarter. It is a global option, so every model on that server gets the same treatment. On a small box with a long window, halving the cache frees more memory than any other single change. Setting num_ctx and what it costs covers the window itself in detail.

What fits on an 8, 16 or 32 GB VPS

Budget weights, plus KV cache, plus headroom for the operating system and anything else running. Two GB of headroom is comfortable on a small VPS.

8 GB. A 4B model at q4_K_M is 2.6 GB and leaves room for a long window. An 8B model at q4_K_M fits with the default 4k window and very little slack. Do not plan on 8B with a 32k window here, because the floor of 10 GB is already past the box.

16 GB. 8B at q4_K_M with a 16k or 32k window is comfortable. 14B at q4_K_M is 9.3 GB of weights and fits with a modest window. 8B at q8_0 is 8.9 GB, so it fits too, and comparing those two on your own prompts is the most useful hour you can spend on this subject.

32 GB. 14B at q8_0 (16 GB) and 32B at q4_K_M (20 GB) both load. The 32B build with a large window will push against the ceiling, so watch ollama ps rather than assuming.

What quantization degrades first

Quantization error does not spread evenly over what a model does. Fluency survives longest, which is exactly why the damage is easy to miss: a badly quantized model still writes clean sentences. Precision goes first. Exact recall of a version number, an API signature or a date. Long chains of reasoning, where a small error at step two becomes a wrong answer at step eight. Strict output formats, where one wrong bracket makes a tool call fail.

That last one is the practical test. When a model has to return JSON your code parses, quantization damage arrives as a parse error rather than as vaguely worse prose, so you see it the same day.

Below four bits the loss gets steep. The q3 and two bit types exist for people squeezing a large model onto small hardware, and they are a real option when the alternative is not running the model at all. They are a poor default. Between q4_K_M and q8_0 the gap is small enough that a published perplexity table will not settle it for your workload, so do not try to settle it that way. Run both against thirty of your own prompts and read the output.

When q8_0 or fp16 is worth the RAM

Pull q8_0 when the memory is genuinely spare and the task punishes small errors: structured extraction, tool calling, code that has to compile. You are buying insurance there, not a noticeably smarter model.

Pull fp16 for two reasons only. Either you are quantizing the model yourself and need the source file, or you are measuring a baseline so you can tell how much your four bit build gave up. Serving from fp16 spends three times the memory of q4_K_M for a difference most people cannot pick out blind, and on a CPU only box it also cuts your token rate to a third.

The stronger rule, at a fixed memory budget: a larger model at q4_K_M usually beats a smaller model at q8_0. 9.3 GB of 14B weights against 8.9 GB of 8B weights is nearly the same RAM (random access memory) and the larger model knows more. Test that on your own prompts rather than taking it on trust.

CPU only inference is limited by memory bandwidth

Most VPS plans have no GPU, so the model runs in system memory on the host CPU. Generation is then bound by memory bandwidth, not by arithmetic, because producing one token requires reading every weight once. That gives a ceiling which has nothing to do with how many cores you bought.

tokens per second ceiling = memory bandwidth / bytes read per token
50 GB/s / 5.2 GB  =  9.6 tokens/s    qwen3 8B q4_K_M
50 GB/s / 8.9 GB  =  5.6 tokens/s    qwen3 8B q8_0
50 GB/s / 16 GB   =  3.1 tokens/s    qwen3 8B fp16

Fifty GB/s is roughly the theoretical figure for a dual channel DDR4-3200 host. Your share is smaller, because a VPS shares that bus with every other tenant on the machine, so treat those numbers as a ceiling nobody reaches. The shape is the useful part: on CPU, halving the bits per weight roughly doubles the token rate. Quantization is the largest speed lever available on a box with no GPU.

Prompt processing behaves differently. Reading a long prompt is compute bound rather than bandwidth bound, so extra cores help there while doing almost nothing for generation speed. A box that ingests a 4k prompt quickly and then generates slowly is behaving normally.

Do not take any of the arithmetic on faith. Measure tokens per second on your own box with the same prompt at each quantization, and let your numbers overrule these.

Quantizing a model yourself

Ollama can build a quantized model from an fp16 or fp32 source, which matters when you have fine tuned something and no library tag exists. Point a Modelfile at the unquantized weights:

FROM /path/to/my/model/f16

Then build and confirm:

ollama create --quantize q4_K_M mymodel
ollama show mymodel

--quantize accepts q8_0, q4_K_S and q4_K_M. There is no q6_K or q5_K_M option here, so for those you quantize with llama.cpp's own tool and import the finished GGUF file. The quantization line from ollama show is how you verify the build did what you asked.

What you will see when it goes wrong

Everything runs on the CPU when you expected the GPU. Read the PROCESSOR column:

ollama ps

It prints 100% GPU, 100% CPU, or a split such as 48%/52% CPU/GPU. A split means the weights plus the KV cache did not fit in VRAM (video RAM, the memory on the graphics card), so part of the model was placed in system memory. Speed then falls close to the CPU only rate, because every token waits on the slow half. Lower the context window, quantize the cache, or pull a smaller build. Adding cores will not help.

The model is killed while loading. Check the kernel and the service log:

sudo dmesg -T | grep -i "out of memory"
sudo journalctl -u ollama -n 50

A line containing Out of memory: Killed process means the total of weights, KV cache and buffers went past the memory on the box. On a VPS with no swap configured, the whole machine can stall for several seconds before that line appears.

Answers got worse and you changed nothing. Two builds of the same model can sit side by side in ollama ls under different tags, and a script that pulls the unsuffixed name will follow whatever the library now points it at. Run ollama show against the exact tag your client requests and read the quantization line, rather than trusting the name in your configuration file.

FAQ

Which Ollama quantization should I pull?

Start with q4_K_M. It is what the Ollama library ships as the default tag for most models, so ollama pull qwen3:8b and ollama pull qwen3:8b-q4_K_M fetch the same file. Move to q8_0 only when the memory is spare and the task punishes small errors, such as tool calling or structured JSON output. When the memory budget is fixed, a larger model at q4_K_M usually beats a smaller model at q8_0, so test that pairing before spending RAM on precision.

Does q4_K_M really mean four bits per weight?

No. Measured on Llama 3.1 8B it is 4.89 bits per weight, because every block of weights stores its own scale and the most sensitive tensors are promoted to a wider type. Q8_0 measures 8.5 bits rather than eight for the same reason. Use the measured figure when estimating: parameter count times bits per weight, divided by eight, gives the file size in bytes.

How much RAM does an 8B model need on a CPU only VPS?

Add weights, KV cache and headroom. Qwen3 8B at q4_K_M is 5.2 GB of weights. At the default 4096 token window the cache adds 0.6 GB, for a floor near 5.8 GB before compute buffers and the operating system. At a 32k window the cache alone is 4.83 GB. Plan on 8 GB for a short window and 16 GB if you want a long one.

Why is my model running at 100% CPU when the box has a GPU?

Run ollama ps and read the PROCESSOR column. 100% CPU, or a split such as 48%/52% CPU/GPU, means the weights plus the KV cache did not fit in VRAM, so Ollama placed part or all of the model in system memory. The usual cause is a context window larger than the card can hold, because the cache is allocated for the entire window when the model loads. Lower the window with OLLAMA_CONTEXT_LENGTH, set OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache, or pull a smaller quantization.