Ollama q4_K_M vs q8_0 vs fp16: which one better?
Use simple arithmetic choose Ollama quantization: see RAM cost for q4_K_M, q8_0 and fp16, plus where quality really drops before you download.
Wetin Ollama quantization dey change
Ollama quantization dey store every weight for model with fewer bits pass the file wey dem take train am. One tag wey end with q4_K_M dey keep about four bits per weight, while fp16 dey keep sixteen, so the download size go be roughly one-third, and the machine go read about one-third of the bytes to produce each token. Dem dey round the weights go coarse grid, dem no dey throw dem away, and for four bits, most models dey answer almost the same way as when dem dey full precision.
Na this be the whole trade-off: memory footprint go small well-well and you go get more tokens per second, but you go lose small accuracy. Wetin follow na how to predict both sides for specific model on top specific box, before you waste twenty minutes dey pull file wey no go fit fit your machine.
If Ollama never start, begin with how to install Ollama for VPS. This page assume say ollama ls don dey work already.
How to read an Ollama quantization tag like q4_K_M
Local models dey ship as GGUF files, wey be the format llama.cpp dey use to store weights for disk. Ollama build on llama.cpp, so Ollama tags carry llama.cpp quantization names unchanged.
The number na the target width. q4 mean say most weight tensors dey packed at four bits each. q8 mean eight. fp16 no be quantized at all: na the model for sixteen bit floating point, wey be the precision most models dey publish with.
K mark a K-quant. Dem group weights into small blocks, and each block store e own scale beside the packed values. Block wey all e weights dey near 0.01 go get fine scale. Block wey hold one large outlier go get coarse one. Na these per-block scales dey make four bit file usable, and dem also be why four bit file no ever be exactly four bits per weight.
The last letter na the mixture. S, M and L decide how many tensors dem promote pass the target width. For q4_K_M, dem store tensors wey rounding dey affect most with wider precision, while most of the rest stay at four bits. Na why q4_K_M dey produce better output than the older q4_0 at almost the same file size.
Ask Ollama wetin e get for disk instead of guessing from the name wey you type:
ollama pull qwen3:8b-q4_K_M
ollama show qwen3:8b-q4_K_Mollama show dey print architecture, parameters, quantization, context length and embedding length. The quantization line na the ground truth for model wey you pull months ago and no longer remember say you choose.
Bits per weight na where file size dey come from
Every size estimate dey start from one number: how many bits the format dey spend per weight, averaged across the whole file. llama.cpp publish measured figures for Llama 3.1 8B for its quantize documentation, and dem work well for any dense model wey get similar shape.
Di data wey dey behind dis chart
[
{
"label": "F16",
"bits_per_weight": 16,
"file_gib": 14.96
},
{
"label": "Q8_0",
"bits_per_weight": 8.5,
"file_gib": 7.95
},
{
"label": "Q6_K",
"bits_per_weight": 6.56,
"file_gib": 6.14
},
{
"label": "Q5_K_M",
"bits_per_weight": 5.7,
"file_gib": 5.33
},
{
"label": "Q4_K_M",
"bits_per_weight": 4.89,
"file_gib": 4.58
},
{
"label": "Q3_K_M",
"bits_per_weight": 3.99,
"file_gib": 3.74
}
]The surprise for that table na the second column. Q4_K_M no be four bits per weight. E measure 4.89 bits, because block scales and promoted tensors dey cost real space too. Q8_0 measure 8.5 bits instead of eight, for the same reason. Use the measured number, and the arithmetic go land within a few percent of the real file:
weight bytes = parameter count x bits per weight / 8
8.03e9 params x 4.89 bits / 8 = 4.91e9 bytes = 4.57 GiBNa the 4.58 GiB Q4_K_M file be this, recovered from two numbers. E also dey close enough to the memory wey the weights occupy after loading. Ollama no unpack anything when e load: the quantized weights dey sit for memory in the same packed form, and each block dey convert as dem use am.
Wetin Ollama dey ship for each model size
The library dey publish q4_K_M, q8_0, and fp16 tag for most families. Some new families no dey follow that pattern; dem dey show for library as cloud-only tags wey no get anything to pull for any width. Na this be the wall wey you go jam when you dey try run GLM 5.2 for VPS. These na the Qwen3 sizes as of October 2026, wey dem read from the tag list for the model page. Every figure wey dey below na disk space before e even touch RAM, and two or three of dem together go full small VPS root volume. So, e good make you know where Ollama dey store the models wey e download before you start to dey collect tags.
Di data wey dey behind dis chart
[
{
"label": "Qwen3 4B",
"q4_K_M_gb": 2.6,
"q8_0_gb": 4.4,
"fp16_gb": 8.1
},
{
"label": "Qwen3 8B",
"q4_K_M_gb": 5.2,
"q8_0_gb": 8.9,
"fp16_gb": 16
},
{
"label": "Qwen3 14B",
"q4_K_M_gb": 9.3,
"q8_0_gb": 16,
"fp16_gb": 30
},
{
"label": "Qwen3 32B",
"q4_K_M_gb": 20,
"q8_0_gb": 35,
"fp16_gb": 66
}
]The default tag get importance here. ollama pull qwen3:8b dey download exactly the same 5.2 GB as ollama pull qwen3:8b-q4_K_M, because the tag wey no get suffix na the q4_K_M build. Q4_K_M no be compromise wey the library just dey manage give people. Na the default wey upstream choose, so to match am na the correct first move for any model wey you never test by yourself. The same reason dey drive the tag choices for running Qwen 3 on a VPS.
The ratios dey constant for every row. To move from q4_K_M go q8_0 go cost you about seventy percent more, e no go be exactly double, because the embedding and output tensors no dey scale the same way as the rest. fp16 na roughly three times q4_K_M. One 32B model for q4_K_M na 20 GB of weights, wey don already pass wetin 16 GB box fit hold if you add any context window at all. For wider view of wetin fit fit which machine, check which models you fit self-host.
Why the KV cache na second cost wey depend on context
Weights na the fixed cost. The KV cache (key and value cache) na the variable one. Every token wey dey inside the context window dey keep im key and value vectors for every layer, so the cache dey grow straight line with the window wey you allow. Dem dey allocate am for the whole window when the model load, no be as the conversation dey fill up, na why long window dey cost you memory even if na one word prompt you send.
KV bytes per token = 2 (key and value) x layers x kv_heads x head_dim x bytes_per_element
Qwen3 8B at f16: 2 x 36 x 8 x 128 x 2 = 147456 bytes = 144 KiB per tokenThose model numbers dey come from the model own configuration: 36 layers, 8 key/value heads, and head dimension of 128. ollama show go give you the architecture and parameter count, and the config.json of the model for Hugging Face go give you the rest. Multiply the cost per token by the window size and the cache no go be small change again.
Di data wey dey behind dis chart
[
{
"label": "4k",
"kv_cache_gb": 0.6,
"floor_ram_gb": 5.8
},
{
"label": "8k",
"kv_cache_gb": 1.21,
"floor_ram_gb": 6.4
},
{
"label": "16k",
"kv_cache_gb": 2.42,
"floor_ram_gb": 7.6
},
{
"label": "32k",
"kv_cache_gb": 4.83,
"floor_ram_gb": 10
}
]For 4096 tokens, the default window wey Ollama dey pick when VRAM dey less than 24 GiB (even for CPU only VPS), the cache dey add 0.6 GB join the weights. If you raise the window go 32k, the cache alone go reach 4.83 GB, wey almost be the same memory as the quantized weights, and the minimum for the whole model go become 10 GB. We dey call am minimum because compute buffers and the operating system still dey top of am. Check the real figure from the SIZE column of ollama ps after the model load. For GPU wey get 24 to 48 GiB of VRAM, the default window na 32k, and from 48 GiB e be 256k, so no assume say the 4k figure go apply for big card.
Na for the server you dey set the window, no be per request, when you run Ollama as service:
OLLAMA_CONTEXT_LENGTH=8192 ollama serveFor systemd install, put am inside drop-in file instead:
sudo systemctl edit ollama[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"Restart with sudo systemctl restart ollama, then check the CONTEXT column of ollama ps to confirm the window wey the running model take load. OLLAMA_KV_CACHE_TYPE dey quantize the cache itself: f16 na the default, q8_0 dey use about half the memory of f16, and q4_0 dey use about one quarter. E be global option, so every model for that server go get the same treatment. Cache quantization only dey work when Flash Attention dey enabled, wey Ollama dey do automatically if the backend support am, and OLLAMA_FLASH_ATTENTION=1 go force am on. For small box wey get long window, to cut the cache for half dey free more memory pass any other single change. If you wan see that saving before you trust am, switching the KV cache from f16 to q8_0 go show you the setting and how e dey drop for ollama ps once the model reload. Setting num_ctx and what it costs explain the window itself well-well. The cache size dey depend on concurrent request slot, no be once per server, so if you allow Ollama answer two prompts at the same time, you go double the figure wey you calculate, na that one be the math behind choosing a parallel slot count and a queue limit.
Wetin fit run for 8, 16, or 32 GB VPS
Budget weights, plus KV cache, plus space wey operating system and anything else wey dey run need. Two GB extra space dey comfortable for small VPS.
8 GB. 4B model for q4_K_M na 2.6 GB, and e leave space for long window. 8B model for q4_K_M fit with default 4k window, but extra space go dey very small. No plan to use 8B with 32k window here, because floor of 10 GB don already pass wetin the machine get.
16 GB. 8B for q4_K_M with 16k or 32k window dey comfortable. 14B for q4_K_M na 9.3 GB of weights, and e fit with modest window. 8B for q8_0 na 8.9 GB, so e fit too. Compare both with your own prompts; na the most useful one hour you fit spend on this matter.
32 GB. 14B for q8_0 (16 GB) and 32B for q4_K_M (20 GB) both fit load. 32B build with large window go push near the memory limit, so monitor ollama ps instead of assuming say e go fit.
Wetin quantization dey spoil first
Quantization error no dey spread equally for all the things wey model dey do. Fluency dey last pass everything, and na why e easy to miss the damage: badly quantized model still fit write clean sentences. Precision dey go first. Exact recall of version number, API signature, or date. Long reasoning chains, where small error for step two fit turn to wrong answer for step eight. Strict output formats, where one wrong bracket fit make tool call fail.
That last one na the practical test. When model must return JSON wey your code go parse, quantization damage go show as parse error instead of prose wey just vaguely worse, so you go see am that same day. Coding agent na the harshest version of this test, because e dey drive model through tool call after tool call. So, pointing agent at your Ollama server go expose over-aggressive quantization within one afternoon.
Below four bits, the loss dey become sharp. The q3 and two-bit types dey for people wey dey squeeze large model onto small hardware. Dem be real option when the alternative na to no run the model at all. But dem no be good default. Between q4_K_M and q8_0, the gap small enough that published perplexity table no go settle which one better for your workload, so no try settle am that way. Run both against thirty of your own prompts and read the output.
When q8_0 or fp16 dey worth the RAM
Pull q8_0 when memory really dey spare and the task no dey tolerate small mistakes: structured extraction, tool calling, and code wey must compile. You dey buy insurance for that case, no be model wey go noticeably smarter.
Pull fp16 for only two reasons. Either you dey quantize the model by yourself and you need the source file, or you dey measure a baseline so you fit know how much your four bit build give up. Serving from fp16 dey use three times the memory of q4_K_M for difference wey most people no fit identify blindly. For CPU only box, e also reduce your token rate to one third.
The stronger rule, with fixed memory budget, be say: bigger model for q4_K_M usually pass smaller model for q8_0. 9.3 GB of 14B weights against 8.9 GB of 8B weights na almost the same RAM (random access memory), and the bigger model sabi more. Test am with your own prompts instead of just trusting am.
CPU-only inference dey limited by memory bandwidth
Most VPS plans no get GPU, so model dey run for system memory on host CPU. Generation come dey bound by memory bandwidth, no be arithmetic, because to produce one token, e need read every weight once. This give one ceiling wey no depend on how many cores you buy.
tokens per second ceiling = memory bandwidth / bytes read per token
50 GB/s / 5.2 GB = 9.6 tokens/s qwen3 8B q4_K_M
50 GB/s / 8.9 GB = 5.6 tokens/s qwen3 8B q8_0
50 GB/s / 16 GB = 3.1 tokens/s qwen3 8B fp16Fifty GB/s na roughly the theoretical figure for dual-channel DDR4-3200 host. Your own share go smaller, because VPS dey share that bus with every other tenant for the machine. So treat those numbers as ceiling wey nobody dey reach. The useful pattern be say: for CPU, if you halve the bits per weight, token rate roughly go double. Quantization na the biggest speed lever wey you get for machine wey no get GPU. Whether the remaining rate dey tolerable depend on the model, and Nemotron 3.5 Lightning for VPS work through that calculation for one specific build, tag, and RAM figure. The other part of the waiting time na how much text the model decide to write. At ten tokens per second, six hundred-token answer go take full minute. So cap the reply with num_predict often fit save more waiting time than another step down for precision.
Prompt processing dey behave different. Reading long prompt dey bound by compute instead of bandwidth, so extra cores help for there but do almost nothing for generation speed. If machine fit ingest 4k prompt quickly and then generate slowly, na normal behaviour.
No just trust any of the arithmetic. Measure tokens per second for your own machine with the same prompt for each quantization, and allow your own numbers override these ones.
Make your own model quantization
Dis one dey important if you don fine-tune something and no library tag dey for am. The way you go follow depend on the format wey you want get for end, because Ollama no dey produce GGUF K-quants wey this page dey measure again.
For K-quant like q4_K_M, quantize am before Ollama see the file. Ollama no dey quantize GGUF models during import, and e go refuse --quantize if you use GGUF source. Quantize the unquantized GGUF wit llama.cpp llama-quantize, then import the file wey you finish. That import way get im own wahala, like chat template mismatch wey fit make the model dey talk anyhow, wey importing a GGUF file into Ollama explain. The quantization line from ollama show na how you go take verify say the file wey you import na the correct build.
Ollama own --quantize dey work only for safetensors imports. Point your Modelfile go the directory wey get the unquantized safetensors weights:
FROM /path/to/safetensors/directoryThen build am:
ollama create --quantize int4 mymodelAs of Ollama v0.40.2 (we check am for 2026-10-11), --quantize dey accept int4, int8, nvfp4, mxfp4 and mxfp8, wit q4 and q8 (plus fp4 and fp8) wey dem dey use as alias for int4 and int8. GGUF names like q4_K_M, q4_K_S, q8_0, q6_K and q5_K_M go show unsupported quantize type error. These types no be the GGUF K-quants wey dey the tables up there, so no use the bits per weight and file size figures for this page take predict wetin go happen for dem.
Wetin you go see when e go wrong
Everything dey run for CPU when you expect GPU. Read the PROCESSOR column:
ollama psE go print 100% GPU, 100% CPU, or split like 48%/52% CPU/GPU. Split mean say the weights plus KV cache no fit inside VRAM (video RAM, the memory for the graphics card), so part of the model enter system memory. Speed go then fall near CPU-only rate, because every token dey wait for the slow half. Reduce the context window, quantize the cache, or use smaller build. Adding more cores no go help.
The model dey get killed while e dey load. Check the kernel and service log:
sudo dmesg -T | grep -i "out of memory"
sudo journalctl -u ollama -n 50Line wey contain Out of memory: Killed process mean say total size of weights, KV cache, and buffers pass the memory wey dey for the box. For VPS wey no get swap configured, the whole machine fit hang for some seconds before that line show.
Answers don worse even though you change nothing. Two builds of the same model fit dey side by side for ollama ls under different tags, and script wey pull the unsuffixed name go follow whichever model the library now point to. Run ollama show against the exact tag wey your client request, then read the quantization line instead of trusting the name for your configuration file.
FAQ
Which Ollama quantization I suppose pull?
Start with q4_K_M. Na wetin Ollama library dey ship as the default tag for most models, so ollama pull qwen3:8b and ollama pull qwen3:8b-q4_K_M go fetch the same file. Move go q8_0 only when memory still plenty and the task no tolerate small errors, like tool calling or structured JSON output. When memory budget fixed, bigger model for q4_K_M usually dey perform better than smaller model for q8_0, so test that pairing before you use RAM for more precision.
q4_K_M really mean four bits per weight?
No. When dem measure am for Llama 3.1 8B, e be 4.89 bits per weight, because every block of weights store its own scale, and the tensors wey sensitive pass dey use wider type. Q8_0 measure 8.5 bits instead of eight for the same reason. Use the measured value when you dey estimate: parameter count times bits per weight, divided by eight, go give the file size for bytes.
How much RAM 8B model need for CPU-only VPS?
Add weights, KV cache, and headroom. Qwen3 8B for q4_K_M get 5.2 GB of weights. For the default 4096 token window, the cache add 0.6 GB, so the minimum dey near 5.8 GB before compute buffers and the operating system. For 32k window, the cache alone na 4.83 GB. Plan for 8 GB if na short window, and 16 GB if you want long one.
Why my model dey use 100% CPU when the box get GPU?
Run ollama ps and read the PROCESSOR column. 100% CPU, or split like 48%/52% CPU/GPU, mean say the weights plus KV cache no fit inside VRAM, so Ollama put part or all of the model for system memory. The usual cause na context window wey pass wetin the card fit hold, because the cache dey allocate for the whole window when the model load. Reduce the window with OLLAMA_CONTEXT_LENGTH, set OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache, or pull smaller quantization.