Ollama q4 vs q8 vs fp16: Which One You Need?

Ollama quantization dey change wetin

Ollama quantization dey store every weight for model with fewer bits than the file wey dem train am with. Tag wey end with q4_K_M dey keep about four bits for each weight, while fp16 dey keep sixteen. So, download go roughly be one-quarter of the size, and machine go read one-quarter as many bytes to produce each token. Dem dey round the weights onto coarse grid; dem no just throw dem away. For most models, four bits still make answers come out close to how dem be for full precision.

Na the whole trade-off be this: memory footprint go much smaller and tokens per second go increase, but accuracy go lose small. The rest of this page show how you fit predict both sides for one specific model on one specific machine, before you spend twenty minutes downloading file wey no go fit.

If Ollama never dey run, start with how to install Ollama for VPS. This page assume say ollama ls don already work.

How to read Ollama quantization tag like q4_K_M

Local models dey come as GGUF files, wey be format llama.cpp dey use to store weights for disk. Ollama build on llama.cpp, so Ollama tags dey carry llama.cpp quantization names unchanged.

The number na the target width. q4 mean say dem pack most weight tensors with four bits each. q8 mean eight bits. fp16 no get quantization at all: na the model for sixteen-bit floating point, wey be the precision most models dey publish with.

K dey mark K-quant. Dem group weights into small blocks, and each block dey store its own scale beside the packed values. Block wey all its weights dey near 0.01 go get fine scale. Block wey hold one large outlier go get coarse scale. Na those per-block scales dey make four-bit file usable, and dem also be why four-bit file no ever be exactly four bits per weight.

The last letter na the mixture. S, M and L decide how many tensors dem promote above the target width. For q4_K_M, dem store tensors wey rounding dey damage pass with wider width, while most of the rest stay for four bits. Na why q4_K_M dey produce better output than the older q4_0 for almost the same file size.

Ask Ollama wetin e get for disk instead of guessing from the name wey you type:

ollama pull qwen3:8b-q4_K_M
ollama show qwen3:8b-q4_K_M

ollama show dey print architecture, parameters, quantization, context length and embedding length. The quantization line na the ground truth for model wey you pull months ago and you no remember again which one you choose.

Bits per weight na where file size dey come from

Every size estimate dey start from one number: how many bits the format dey use per weight, averaged across the whole file. llama.cpp publish measured figures for Llama 3.1 8B for its quantize documentation, and dem work well for any dense model wey get similar shape.

[
  {
    "label": "F16",
    "bits_per_weight": 16,
    "file_gib": 14.96
  },
  {
    "label": "Q8_0",
    "bits_per_weight": 8.5,
    "file_gib": 7.95
  },
  {
    "label": "Q6_K",
    "bits_per_weight": 6.56,
    "file_gib": 6.14
  },
  {
    "label": "Q5_K_M",
    "bits_per_weight": 5.7,
    "file_gib": 5.33
  },
  {
    "label": "Q4_K_M",
    "bits_per_weight": 4.89,
    "file_gib": 4.58
  },
  {
    "label": "Q3_K_M",
    "bits_per_weight": 3.99,
    "file_gib": 3.74
  }
]

The surprise for that table na the second column. Q4_K_M no be four bits per weight. E measure 4.89 bits, because block scales and promoted tensors too dey use real space. Q8_0 measure 8.5 bits instead of eight, for the same reason. Use the measured number and the calculation go land within few percent of the real file:

weight bytes = parameter count x bits per weight / 8
8.03e9 params x 4.89 bits / 8 = 4.91e9 bytes = 4.57 GiB

Na the 4.58 GiB Q4_K_M file be that, recovered from two numbers. E also dey close enough to the memory wey the weights occupy after dem load am. Ollama no unpack anything during load: the quantized weights dey stay for memory for the same packed form, and dem dey convert each block as dem use am.

Wetin Ollama really dey release for each model size

Library dey publish q4_K_M, q8_0 and fp16 tag for most families. These na the Qwen3 sizes as of August 2026, wey dem read from the tag list for the model page.

[
  {
    "label": "Qwen3 4B",
    "q4_K_M_gb": 2.6,
    "q8_0_gb": 4.4,
    "fp16_gb": 8.1
  },
  {
    "label": "Qwen3 8B",
    "q4_K_M_gb": 5.2,
    "q8_0_gb": 8.9,
    "fp16_gb": 16
  },
  {
    "label": "Qwen3 14B",
    "q4_K_M_gb": 9.3,
    "q8_0_gb": 16,
    "fp16_gb": 30
  },
  {
    "label": "Qwen3 32B",
    "q4_K_M_gb": 20,
    "q8_0_gb": 35,
    "fp16_gb": 66
  }
]

The default tag matter for here. ollama pull qwen3:8b dey download exactly the same 5.2 GB like ollama pull qwen3:8b-q4_K_M, because the tag wey no get suffix na the q4_K_M build. Q4_K_M no be compromise wey library reluctantly dey offer. Na the default wey upstream choose, so matching am na the sensible first step for any model wey you never test by yourself. Na the same reasoning drive the tag choices for running Qwen 3 for VPS.

The ratios dey hold for every row. If you move from q4_K_M go q8_0, e go cost about seventy percent more instead of exactly double, because the embedding and output tensors no dey scale the same way like the rest. fp16 dey roughly three times q4_K_M. A 32B model for q4_K_M get 20 GB of weights, and this don already pass wetin 16 GB box fit hold with any context window at all. For wider view of which machine fit run which model, see which models you fit self-host.

Wetin make KV cache be a second cost wey depend on context

Weights na the fixed cost. KV cache (key and value cache) na the variable cost. Every token for the context window keep its key and value vectors for every layer, so the cache dey grow in a straight line with the window size wey you allow. The system allocate am for the whole window when the model load, no be as the conversation dey fill up. Na why long window still dey use memory even when prompt na just one word.

KV bytes per token = 2 (key and value) x layers x kv_heads x head_dim x bytes_per_element
Qwen3 8B at f16:  2 x 36 x 8 x 128 x 2 = 147456 bytes = 144 KiB per token

Those model numbers come from the model own configuration: 36 layers, 8 key/value heads, and head dimension of 128. ollama show give you the architecture and parameter count, while the model config.json for Hugging Face give you the remaining details. Multiply the cost per token by the window size, and the cache no go remain small enough to ignore.

[
  {
    "label": "4k",
    "kv_cache_gb": 0.6,
    "floor_ram_gb": 5.8
  },
  {
    "label": "8k",
    "kv_cache_gb": 1.21,
    "floor_ram_gb": 6.4
  },
  {
    "label": "16k",
    "kv_cache_gb": 2.42,
    "floor_ram_gb": 7.6
  },
  {
    "label": "32k",
    "kv_cache_gb": 4.83,
    "floor_ram_gb": 10
  }
]

For Ollama default window of 4096 tokens, the cache add 0.6 GB on top of the weights. If you increase the window to 32k, the cache alone reach 4.83 GB. That one almost match the memory wey the quantized weights use, and the minimum memory for the whole model become 10 GB. We call am a floor because compute buffers and the operating system still need memory on top. Read the actual figure from the SIZE column of ollama ps after the model load.

The server set the window, no be each request, when you run Ollama as a service:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

For systemd install, put am inside a drop-in instead:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Restart with sudo systemctl restart ollama. Then check the CONTEXT column of ollama ps to confirm the window wey the running model actually load with. OLLAMA_KV_CACHE_TYPE fit quantize the cache itself: f16 na the default, q8_0 use about half the memory of f16, and q4_0 use about one-quarter. Na global option this one be, so every model for that server get the same setting. For small box wey get long window, cutting the cache by half free more memory than any other single change. Setting num_ctx and wetin e cost explain the window itself in detail.

Wetin fit run for 8, 16 or 32 GB VPS

Budget weights, plus KV cache, plus space wey operating system and anything else wey dey run need. Two GB of extra space dey comfortable for small VPS.

8 GB. 4B model for q4_K_M na 2.6 GB, and e still leave space for long window. 8B model for q4_K_M fit with the default 4k window, but extra space go dey very small. No plan to use 8B with 32k window for here, because the minimum of 10 GB don already pass the capacity of the machine.

16 GB. 8B for q4_K_M with 16k or 32k window dey comfortable. 14B for q4_K_M na 9.3 GB of weights, and e fit with modest window. 8B for q8_0 na 8.9 GB, so e fit too. Compare the two with your own prompts; na the most useful hour you fit spend on this matter.

32 GB. 14B for q8_0 (16 GB) and 32B for q4_K_M (20 GB) both fit load. 32B build with large window go push close to the memory limit, so monitor ollama ps instead of assuming say e go fit.

Wetin quantization dey spoil first

Quantization error no dey affect everything wey model dey do the same way. Fluency dey last pass, and na why damage easy to miss: badly quantized model still fit write clean sentences. Precision dey go first. Exact recall of version number, API signature, or date. Long reasoning chains, where small error for step two turn to wrong answer for step eight. Strict output formats, where one wrong bracket fit make tool call fail.

Na that last one be the practical test. When model suppose return JSON wey your code dey parse, quantization damage go show as parse error instead of prose wey just vaguely worse, so you go see am that same day. Coding agent na the hardest version of this test, because e dey drive model through tool call after tool call, so pointing agent at your Ollama server go expose over-aggressive quantization within one afternoon.

Below four bits, the loss dey become steep. The q3 and two-bit types exist for people wey dey squeeze large model onto small hardware, and dem na real option when the alternative na not to run the model at all. Dem no good as default. Between q4_K_M and q8_0, the gap small enough say published perplexity table no go settle am for your workload, so no try settle am that way. Run both against thirty of your own prompts and read the output.

When q8_0 or fp16 worth the RAM

Pull q8_0 only when memory really dey spare and task no dey tolerate small mistakes: structured extraction, tool calling, or code wey must compile. Na insurance you dey buy there, no be model wey go look noticeably smarter.

Pull fp16 for only two reasons. Either you dey quantize the model yourself and need the source file, or you dey measure a baseline so you fit know how much your four bit build lose. Serving from fp16 dey use three times the memory of q4_K_M for difference wey most people no fit identify blindly. For CPU-only box, e also cut your token rate to one third.

The stronger rule for fixed memory budget be say: bigger model for q4_K_M usually beat smaller model for q8_0. 9.3 GB of 14B weights against 8.9 GB of 8B weights na almost the same RAM (random access memory), and the bigger model know more. Test am with your own prompts instead of just trusting am.

CPU-only inference dey limited by memory bandwidth

Most VPS plans no get GPU, so model dey run for system memory on host CPU. Generation dey limited by memory bandwidth, no be arithmetic, because to produce one token, system need read every weight once. This create one ceiling wey no depend on how many cores you buy.

tokens per second ceiling = memory bandwidth / bytes read per token
50 GB/s / 5.2 GB  =  9.6 tokens/s    qwen3 8B q4_K_M
50 GB/s / 8.9 GB  =  5.6 tokens/s    qwen3 8B q8_0
50 GB/s / 16 GB   =  3.1 tokens/s    qwen3 8B fp16

Fifty GB/s na roughly the theoretical figure for dual-channel DDR4-3200 host. Your own share go smaller, because VPS dey share that bus with every other tenant for the machine. So treat those numbers as ceiling wey nobody dey reach. The useful part na the pattern: for CPU, if you halve the bits per weight, token rate go roughly double. Quantization na the biggest speed lever wey you get for machine wey no get GPU.

Prompt processing dey behave differently. E dey limited by compute instead of bandwidth, so extra cores dey help there but dem do almost nothing for generation speed. If machine dey process 4k prompt quickly and then generate slowly, na normal behaviour.

No accept any of the arithmetic without checking am. Measure tokens per second for your own machine with the same prompt for each quantization, then make your numbers overrule these ones.

Model quantize by yourself

Ollama fit build quantized model from fp16 or fp32 source. This dey matter when you fine-tune something and no library tag dey available. Point a Modelfile to the unquantized weights:

FROM /path/to/my/model/f16

Then build am and confirm:

ollama create --quantize q4_K_M mymodel
ollama show mymodel

--quantize accepts q8_0, q4_K_S and q4_K_M. No q6_K or q5_K_M option dey here. So, for those ones, use llama.cpp own tool to quantize am, then import the finished GGUF file. The quantization line from ollama show na how you verify say the build do wetin you request.

Wetin you go see when e spoil

Everything dey run for CPU when you expect GPU. Read the PROCESSOR column:

ollama ps

E go print 100% GPU, 100% CPU, or split like 48%/52% CPU/GPU. Split mean say the weights plus KV cache no fit inside VRAM (video RAM, the memory wey dey for graphics card), so dem put part of the model for system memory. Speed go then drop near CPU-only rate, because every token dey wait for the slower part. Reduce the context window, quantize the cache, or use smaller build. Adding more cores no go help.

The model dey get killed while e dey load. Check the kernel and service log:

sudo dmesg -T | grep -i "out of memory"
sudo journalctl -u ollama -n 50

Line wey contain Out of memory: Killed process mean say the total size of weights, KV cache, and buffers pass the memory wey dey for the machine. For VPS wey no get swap configured, the whole machine fit stall for several seconds before the line show.

Answers don worse even though you change nothing. Two builds of the same model fit dey side by side for ollama ls under different tags, and script wey pull the unsuffixed name go follow wherever the library currently point am. Run ollama show against the exact tag wey your client request, then read the quantization line instead of trusting the name for your configuration file.

FAQ

Which Ollama quantization should I pull?

Start with q4_K_M. Na the tag wey Ollama library dey ship as default for most models, so ollama pull qwen3:8b and ollama pull qwen3:8b-q4_K_M go fetch the same file. Move to q8_0 only when memory dey spare and the task no tolerate small errors, like tool calling or structured JSON output. When memory budget fixed, bigger model for q4_K_M usually pass smaller model for q8_0, so test that pairing before you use RAM for more precision.

q4_K_M really mean four bits per weight?

No. For Llama 3.1 8B measurement, e be 4.89 bits per weight, because every block of weights dey store its own scale and dem promote the most sensitive tensors to wider type. Q8_0 measure 8.5 bits instead of eight for the same reason. Use the measured figure for estimation: parameter count times bits per weight, divide by eight, go give you the file size in bytes.

How much RAM 8B model need for CPU-only VPS?

Add weights, KV cache, and headroom together. Qwen3 8B for q4_K_M get 5.2 GB weights. For the default 4096 token window, the cache add 0.6 GB, so the minimum dey near 5.8 GB before compute buffers and the operating system. For 32k window, the cache alone na 4.83 GB. Plan for 8 GB if na short window, and 16 GB if you want long one.

Why my model dey use 100% CPU when the box get GPU?

Run ollama ps and read the PROCESSOR column. 100% CPU, or split like 48%/52% CPU/GPU, mean say the weights plus KV cache no fit inside VRAM, so Ollama put part or all of the model for system memory. The usual cause na context window wey bigger than wetin the card fit hold, because the cache dey allocate for the whole window when the model load. Reduce the window with OLLAMA_CONTEXT_LENGTH, set OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache, or pull smaller quantization.