Ollama KV cache quantization: q8_0 vs f16

What grew when you raised num_ctx

Ollama KV cache quantization is one server setting, OLLAMA_KV_CACHE_TYPE, with three documented values: f16 (the default), q8_0 and q4_0. It only works when flash attention is on, and it applies to every model the server loads. You set it as an Environment= line on the ollama systemd unit. The Ollama FAQ states the memory effect: q8_0 uses approximately half the memory of f16, and q4_0 approximately a quarter. Everything below was checked against that FAQ and the Ollama 0.34.2 release in September 2026.

The reason you are reading this is usually the same. You picked a weight quantization that fits, then raised num_ctx, and the model stopped fitting, or started swapping. The weights did not change size. The KV cache did, because it grows with context length, and at 32k or 64k tokens it can be bigger than the saving you made by picking q4 weights instead of q8.

How big is the KV cache

KV stands for key and value. For every token in the context, each attention layer stores one key vector and one value vector, so later tokens can attend to it. That store is the KV cache. It is a different thing from the prompt cache that lets a repeated prefix skip prefill; the KV cache and the prompt cache are often confused, and only the first one costs you memory in proportion to num_ctx.

The size is arithmetic. Per token, in bytes:

2 (K and V) x layers x KV heads x head dimension x bytes per value

With f16, one value is 2 bytes. Read the other numbers from the model itself:

ollama show llama3.1:8b --verbose | grep -E 'block_count|head_count|embedding_length'

For llama3.1:8b that prints lines containing llama.block_count 32, llama.attention.head_count 32, llama.attention.head_count_kv 8 and llama.embedding_length 4096. The head dimension is embedding_length / head_count, so 4096 / 32 = 128. The KV heads number 8, not 32, because the model uses grouped-query attention (GQA), where several query heads share one key and value head. That alone makes the cache four times smaller than it would be on an older architecture.

So one token costs 2 x 32 x 8 x 128 x 2 = 131,072 bytes, which is 128 KiB. Multiply by the context, and the chart below is what you get. These are computed from the formula, not measured, and they are for the cache alone. The weights and the compute buffers come on top.

[
  {
    "label": "4096",
    "f16_gib": 0.5,
    "q8_0_gib": 0.27,
    "q4_0_gib": 0.14
  },
  {
    "label": "8192",
    "f16_gib": 1.0,
    "q8_0_gib": 0.53,
    "q4_0_gib": 0.28
  },
  {
    "label": "16384",
    "f16_gib": 2.0,
    "q8_0_gib": 1.06,
    "q4_0_gib": 0.56
  },
  {
    "label": "32768",
    "f16_gib": 4.0,
    "q8_0_gib": 2.13,
    "q4_0_gib": 1.13
  },
  {
    "label": "65536",
    "f16_gib": 8.0,
    "q8_0_gib": 4.25,
    "q4_0_gib": 2.25
  },
  {
    "label": "131072",
    "f16_gib": 16.0,
    "q8_0_gib": 8.5,
    "q4_0_gib": 4.5
  }
]

At a num_ctx of 32768 the f16 cache is 4.0 GiB. Doubling to 65536 doubles it to 8.0 GiB, and the model's full 131072 context needs 16.0 GiB for the cache before a single weight is loaded. The q8_0 and q4_0 columns sit a little above half and a quarter because of how the quantized blocks are laid out: every 32 values share one 16-bit scale, so a q8_0 block is 34 bytes and a q4_0 block is 18 bytes, against 64 bytes for f16. That is 53 percent and 28 percent, which is what the FAQ rounds to one half and one quarter.

Two things the formula does not tell you. Models with sliding-window layers, such as the Gemma family, allocate a much smaller cache for those layers, so the formula overestimates them. And OLLAMA_NUM_PARALLEL multiplies the whole thing, because Ollama starts the runner with a context of num_ctx times the parallel slot count. The default is one slot, so this only bites if you raised it.

Why doubling num_ctx cost more than the weight quantization saved

Put the two numbers side by side for the same model. The Ollama library lists llama3.1:8b-instruct-q4_K_M at 4.9 GB and llama3.1:8b-instruct-q8_0 at 8.5 GB (September 2026, rounded file sizes, converted to GiB below).

[
  {
    "label": "Weights, q4_K_M",
    "gib": 4.56
  },
  {
    "label": "Weights, q8_0",
    "gib": 7.92
  },
  {
    "label": "KV cache f16, num_ctx 32768",
    "gib": 4.0
  },
  {
    "label": "KV cache f16, num_ctx 65536",
    "gib": 8.0
  }
]

Moving the weights from q8_0 (7.92 GiB) to q4_K_M (4.56 GiB) saved about 3.4 GiB. Moving num_ctx from 32768 to 65536 at f16 took the cache from 4.0 GiB to 8.0 GiB, an extra 4 GiB. The context step cost more than the weight step saved. That is the whole trap, and it is why working out whether a model fits in your RAM has to include the context you intend to run as well as the file size on the library page.

The KV cache type is the lever that acts on the right-hand bars. Weight quantization does nothing to them.

Measure the baseline before you change anything

Do this on the server with a model already pulled. Load it with a large context through the API, then look at what Ollama reports. The ollama CLI is only a client of the server on port 11434, so the request below is what ollama run would send anyway, without the interactive session.

curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.1:8b","prompt":"Say hello.","stream":false,"options":{"num_ctx":32768}}' > /dev/null
ollama ps

ollama ps prints one row per loaded model with the columns NAME, ID, SIZE, PROCESSOR, CONTEXT and UNTIL. Confirm CONTEXT reads 32768; if it shows a smaller number the option was not applied and the rest of the test is meaningless. Write down SIZE. That is Ollama's estimate of everything the model occupies, weights and cache together. On a VPS without a GPU, PROCESSOR reads 100% CPU, which is expected and is not a problem.

Now read the allocation itself. Ollama 0.34 runs GGUF models through llama.cpp's llama-server and forwards its log lines into the journal, and llama.cpp prints the cache size when it allocates it.

sudo journalctl -u ollama --since "5 min ago" --no-pager | grep -E 'starting llama-server|llama_kv_cache: size|Flash Attention'

The starting llama-server line is the full command Ollama ran, including -c 32768, --flash-attn auto and, once you have set it, --cache-type-k and --cache-type-v. The cache line for llama3.1:8b at this context should read, give or take the prefix and the spacing:

llama_kv_cache: size = 4096.00 MiB ( 32768 cells,  32 layers,  1/1 seqs), K (f16): 2048.00 MiB, V (f16): 2048.00 MiB

That matches the arithmetic above exactly: 128 KiB per token times 32768 tokens. K (f16) and V (f16) are the parts you are about to shrink. Note the Flash Attention line too. With --flash-attn auto, llama.cpp probes the backend and logs whether it ended up enabled, and the next step depends on that answer.

Set OLLAMA_KV_CACHE_TYPE with systemctl edit

The Ollama installer creates a systemd unit, so the setting goes in a drop-in for that unit. The FAQ's instruction is exactly this. Set flash attention explicitly at the same time, because a quantized cache without it is refused by the runner, and auto leaves the decision to a probe you did not see.

sudo systemctl edit ollama.service

An editor opens on a temporary file full of comments. Between the marker lines, above the one that says edits below it will be discarded, add:

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Save and quit. If you prefer a file to an editor, the same three lines in /etc/systemd/system/ollama.service.d/override.conf do the same job, which is what systemctl edit writes for you. Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl show ollama -p Environment

The last command should print one Environment= line holding the installer's PATH= plus your two variables. If they are missing, the drop-in did not take, and the section on failures below covers why. Restarting the service unloads every model, so this is a change to make between sessions, not during one.

Ollama also logs its full configuration on every start, so the second confirmation is in the journal:

sudo journalctl -u ollama -n 200 --no-pager | grep -o 'OLLAMA_KV_CACHE_TYPE:[a-z0-9_]*'

OLLAMA_KV_CACHE_TYPE:q8_0 means the server process has the variable. An empty result means it does not, whatever your shell says.

Check the effect in ollama ps and the runner log

Repeat the baseline exactly: the same model and the same num_ctx, with the same commands.

curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.1:8b","prompt":"Say hello.","stream":false,"options":{"num_ctx":32768}}' > /dev/null
ollama ps
sudo journalctl -u ollama --since "5 min ago" --no-pager | grep -E 'starting llama-server|llama_kv_cache: size'

ollama ps should show a smaller SIZE at the same CONTEXT. The starting llama-server line should now carry --cache-type-k q8_0 --cache-type-v q8_0 and --flash-attn on. And the cache line should have changed type and size:

llama_kv_cache: size = 2176.00 MiB ( 32768 cells,  32 layers,  1/1 seqs), K (q8_0): 1088.00 MiB, V (q8_0): 1088.00 MiB

That is 2.13 GiB against 4.0 GiB before, the 53 percent from the block arithmetic. The saving is what lets you raise num_ctx again, or fit a larger weight file, on the same box. The log line is the allocation. ollama ps is an estimate of the whole model and is the number the scheduler uses to decide whether a model fits at all, so watch both, and trust the log line when they disagree.

If you want the speed side of the trade, run the same prompt with ollama run llama3.1:8b --verbose before and after and compare the eval rate it prints. Flash attention changes the attention kernel, and quantized cache values must be dequantized on every read, so the tokens-per-second figure can move in either direction on a CPU box. Measuring tokens per second properly is a few minutes of work and is the only honest answer for your hardware.

The quality trade, and why it is not the same as weight quantization

The FAQ describes q8_0 as "a very small loss in precision" and q4_0 as "a small-medium loss in precision that may be more noticeable at higher context sizes". Take the second half of that sentence seriously. Weight quantization and KV cache quantization are both called quantization, but they are different operations with different failure modes.

Weights are quantized once, offline, by whoever built the GGUF. The K in q4_K_M marks a scheme that spends more bits on the tensors that matter most, and the error it introduces is fixed at build time, so it can be measured once against the original before anyone runs the model.

The KV cache is quantized at run time, on activations, in blocks of 32 values with one shared scale, with no calibration and no per-tensor choices. Every token you generate attends to every earlier key and value through that rounding, so a longer context means more rounded values in every attention step. That is why the FAQ says the loss is more noticeable at higher context sizes, and higher context sizes are the only reason you are here. A 4-bit cache at 4k tokens and a 4-bit cache at 64k tokens are not the same risk.

Ollama sets the key cache and the value cache to the same type; there is no way to quantize one and not the other through this setting. So the practical rule is: q8_0 is the setting you leave on. q4_0 is the setting you test for your task before you trust it. A cheap test is a long document with one specific fact planted in the middle, a question that needs that fact, and the same request at f16 and q4_0 with "temperature": 0 in the options. If the answers diverge, that is your task's verdict, and no general benchmark overrides it. Set the cache type back with the same two commands, daemon-reload and restart.

None of this affects the weights. If the model was wrong with an f16 cache, a different cache type will not fix it, and if it fits only with a q4_0 cache the honest options are a smaller weight file, a smaller num_ctx, more RAM, or a different model.

What breaks, with the strings you will see

The model refuses to load after the change. The runner exits and journalctl -u ollama holds:

quantized V cache was requested, but this requires Flash Attention

llama.cpp resolved flash attention to disabled, so it will not allocate a quantized value cache. Set OLLAMA_FLASH_ATTENTION=1 explicitly rather than leaving it on auto, then reload and restart. If the same line comes back with it forced on, the backend for your hardware does not support the attention kernel, and the cache has to stay f16 on that box.

Unsupported cache type: q8. The value is passed to the runner as a string and matched exactly. Ollama lowercases it for you, so Q8_0 works, but q8 and int8 do not. Use f16, q8_0 or q4_0, which are the three the Ollama documentation lists.

Editing "/etc/systemd/system/ollama.service.d/override.conf" canceled: temporary file is empty. You typed the lines below the marker that says everything after it is discarded. Run systemctl edit again and put them above it, or write the override file directly.

Warning: The unit file, source configuration file or drop-ins of ollama.service changed on disk. Run 'systemctl daemon-reload' to reload units. Printed by systemctl restart when you skipped daemon-reload. The restart used the old environment. Reload, then restart again.

Nothing changed, and the journal never shows the variable. You set it with export in your shell before running ollama run. That reaches the CLI, which is a client, and never reaches the server, which is the systemd unit with its own environment. systemctl show ollama -p Environment is the only view that counts.

SIZE is still far bigger than the weights. Read -c and -np on the starting llama-server line. If -c is a multiple of your num_ctx, you have more than one parallel slot, and every slot gets a full cache. Also check that CONTEXT in ollama ps is what you asked for; a client that sets its own num_ctx overrides yours, and how Ollama picks the context length has more than one input.

The model unloads between your two measurements. The default keep-alive is five minutes, so a slow second command can hit an empty ollama ps. Pass "keep_alive": "30m" in the request, or keep the model loaded for the whole session.

FAQ

Does OLLAMA_KV_CACHE_TYPE work without flash attention?

No. The runner refuses a quantized value cache when flash attention is off and logs quantized V cache was requested, but this requires Flash Attention. Ollama 0.34 sets flash attention to auto and lets llama.cpp probe the backend, so set OLLAMA_FLASH_ATTENTION=1 in the same drop-in as the cache type and you remove the guess. If it still fails with it forced on, your backend has no flash attention kernel and the cache stays f16.

Is q8_0 KV cache safe to leave on all the time?

The Ollama FAQ calls its precision loss "very small", and its memory is about 53 percent of f16 by the block layout, so for most servers it is the right default. The setting is global, so every model the server loads gets it. Test it once on your own task with "temperature": 0 at f16 and at q8_0, and if the outputs match you are done.

Can I set the KV cache type per model?

Not with this setting. The documentation states it is a global option that applies to every model the server runs. The controls you do have per model are num_ctx, which is what the cache size scales with, and the weight file you pull. If one model needs an f16 cache and another does not, the honest answer is a second Ollama instance on a different port with its own drop-in.

Why did raising num_ctx use more memory than the weights did?

Because the KV cache is proportional to the context length while the weights are fixed. For llama3.1:8b the cache costs 128 KiB per token at f16, which is 4 GiB at 32768 tokens and 8 GiB at 65536. Dropping the weights from q8_0 to q4_K_M saves about 3.4 GiB, so one doubling of the context undoes the whole saving. Run ollama show <model> --verbose, multiply 2 x layers x KV heads x head dimension x 2 bytes for your own model, or read the llama_kv_cache: size line out of the journal.

What should I set on a CPU-only VPS?

The same two lines: OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0. The memory arithmetic is identical whether the cache lives in RAM or VRAM, and on a VPS RAM is the whole budget. Confirm the Flash Attention line in the journal shows it enabled, compare ollama ps at the same CONTEXT before and after, and measure tokens per second yourself rather than assuming it went up.