Ollama context length: setting num_ctx

What num_ctx does, and why your long prompt got cut

The Ollama context length is the number of tokens a loaded model can hold in memory at once, and num_ctx is the option that sets it. Ollama picks a default that is far below the maximum the model advertises, so a longer prompt is cut before the model ever reads it. Nothing in the response tells you it happened.

Llama 3.1 8B is listed with a 128k context window on the Ollama model library. A stock server will not give you that. Ollama's own documentation carries different defaults on different pages: the FAQ says 4096 tokens, the Modelfile reference says num_ctx defaults to 2048, and the context length page says the default is picked from available VRAM (video RAM), 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k above that. Each was true of some build. That disagreement is the useful lesson here: read the value off your own running server instead of trusting any page, including this one.

Truncation is quiet because the model still answers, and the answer still reads well. It was written from the tail of your input. A summary that misses the first half of a document looks like a weak model. It is usually a small context window.

Check the Ollama context length your server actually applied

The check that works on any build is prompt_eval_count, the count of prompt tokens the server reports it processed. Send more than the context can hold and that number stops at the limit.

sudo apt update && sudo apt install -y jq
LONG=$(python3 -c "print('the quick brown fox jumps over the lazy dog. ' * 2000)")
jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:4096}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{prompt_eval_count, prompt_eval_duration}'

That prompt is about 18,000 words, so far more than 4096 tokens. prompt_eval_count comes back near 4096 rather than near the real token count, because the server dropped the rest. Run it again with "num_ctx":16384 and the count rises. If your build returns an error instead of truncating, that is the same finding with a louder signal.

ollama ps

The CONTEXT column, on builds that print it, holds the context length the loaded model is running with right now. The PROCESSOR column next to it shows where the model sits. 100% CPU is normal on a VPS with no GPU. A split such as 30%/70% CPU/GPU on a GPU box means the weights plus the cache no longer fit in VRAM, and a raised num_ctx is the usual reason.

journalctl -u ollama --no-pager | grep -i n_ctx | tail -n 5

The inference runner prints its context size in a line containing n_ctx. The exact wording moves between releases, so treat a missing line as a rename rather than as proof of anything.

Four places to set num_ctx

In the request. Send "options": {"num_ctx": 16384} to /api/generate or /api/chat. This beats every other setting, and it applies to that one call. If the value differs from what the loaded model is running with, the server reloads the model first, which you can see in load_duration in the response: it jumps from near zero to whole seconds.

In the interactive session. Inside ollama run, type /set parameter num_ctx 16384. It lasts for that session.

In a Modelfile. This bakes the value into a named model, so every client gets it with no client-side change.

FROM llama3.1:8b
PARAMETER num_ctx 16384

ollama create llama3.1-16k -f ./Modelfile
ollama run llama3.1-16k

On the server. OLLAMA_CONTEXT_LENGTH sets the default for every request that does not carry its own num_ctx. Under systemd, add a drop-in instead of editing the unit file.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"

sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama ps

The precedence matters most when you are debugging somebody else's client. A request that carries num_ctx beats the server default, so a chat front end or an agent that sends a small value of its own quietly undoes your systemd change. When you point a coding agent at your Ollama server, check what the client sends before you blame the server.

Why you cannot just set num_ctx to the model maximum

Attention makes every token look at every token before it. The keys and values computed for earlier tokens are kept so they are not recomputed for each new token, and that store is the KV cache (key/value cache). It is allocated for the whole of num_ctx when the model loads, not as the conversation grows, so a large context costs its memory even on a one line prompt.

DigitalOcean's inference cost tutorial states the arithmetic in one line:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value

The 2 counts keys and values separately. Read the other numbers off your own model.

curl -s http://localhost:11434/api/show -d '{"model":"llama3.1:8b"}' |
  jq '.model_info | {ctx: ."llama.context_length", layers: ."llama.block_count", heads: ."llama.attention.head_count", kv_heads: ."llama.attention.head_count_kv", embed: ."llama.embedding_length"}'

Llama 3.1 8B reports 32 layers and 8 key/value heads. The head dimension is embed divided by heads, so 4096 / 32 = 128 here, and some models publish it directly as llama.attention.key_length. The default cache holds f16 values, so bytes_per_value is 2, and 2 32 8 128 2 works out at 131,072 bytes. That is 128 KiB of cache for every single token of context. Multiply by the context length and the cost stops being abstract.

[
  {
    "label": "4k",
    "kv_cache_gib": 0.5,
    "total_ram_gib": 5.1
  },
  {
    "label": "8k",
    "kv_cache_gib": 1,
    "total_ram_gib": 5.6
  },
  {
    "label": "16k",
    "kv_cache_gib": 2,
    "total_ram_gib": 6.6
  },
  {
    "label": "32k",
    "kv_cache_gib": 4,
    "total_ram_gib": 8.6
  },
  {
    "label": "64k",
    "kv_cache_gib": 8,
    "total_ram_gib": 12.6
  },
  {
    "label": "128k",
    "kv_cache_gib": 16,
    "total_ram_gib": 20.6
  }
]

Those 6 rows are arithmetic from the formula above, not measurements. The total column adds the 4.9 GB download that the Ollama library listed for llama3.1:8b in August 2026, which is 4.6 GiB, and it leaves out the compute buffers and the server process itself. Treat it as a floor.

The shape is the point. At 8k the cache costs 1 GiB, which is noise next to the weights. At the model's full 128k it costs 16 GiB, more than three times the weights, for a total near 20.6 GiB. So a 4 GB VPS cannot load this model at any useful context. An 8 GB VPS is comfortable at 8k. A 16 GB VPS reaches 32k with room left for the rest of the box.

What happens when the KV cache does not fit

On a CPU-only VPS the process simply grows. Watch it while the model loads and while a long request runs.

free -m
ps -eo rss,comm --sort=-rss | head -n 5

RSS (resident set size) is printed in kilobytes. If used swap in free -m starts climbing, back the context off. A KV cache that lives in swap makes generation stall for seconds per token, because each new token reads the whole cache.

If the box runs out of memory completely, the kernel picks the largest process and kills it.

sudo dmesg | grep -i "killed process"

A line reading Out of memory: Killed process 1234 (ollama) means the context you asked for did not fit. Ollama often refuses before it gets that far, and the request then fails with a message naming the memory it wanted against the memory that was free.

On a GPU box the failure is quieter. Layers spill into system RAM, ollama ps shows the CPU and GPU split, and throughput drops sharply. How sharply depends on your hardware, so measure tokens per second on your own box at each context setting rather than trusting a figure from someone else's machine.

Prefill time grows faster than the prompt

Prefill is the work done on your input before the first output token appears. Each prompt token attends to every token before it, so the total work grows with the square of the input length. Doubling the prompt more than doubles the wait for the first token.

The response carries the measurement, so you do not have to take that on trust.

jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:16384}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{tokens: .prompt_eval_count, prefill_seconds: (.prompt_eval_duration/1000000000)}'

Run that with a short prompt and again with a long one, then divide tokens by seconds in each case. On a CPU-only VPS, prefill is usually the slowest part of a long-context request, and a tokens per second figure taken from a short prompt will not predict it.

Concurrency is where this hurts most. Every request being served needs its own cache, so the memory in the chart above is per request rather than per server, and one long request can hold the box while short ones queue behind it. Set OLLAMA_NUM_PARALLEL deliberately, and read how many concurrent users one self-hosted LLM can serve before you raise both numbers together.

Buy context back with a smaller cache

bytes_per_value in the formula is a setting you control. Ollama's FAQ documents OLLAMA_KV_CACHE_TYPE, with f16 as the default at 2 bytes, plus q8_0 at 1 byte and q4_0 below that. Moving to q8_0 halves the cache, so the 32k row costs 2 GiB instead of 4 GiB. The same FAQ documents OLLAMA_FLASH_ATTENTION=1, which some builds want before a quantised cache takes effect.

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Confirm rather than assume: restart the service, load the model at the same num_ctx as before, and compare RSS. Support depends on the model and on the backend, so a setting that changes nothing means your combination is not covered. The documentation lists these options without promising a quality result, so test q4_0 against your own prompts before you rely on it. If these knobs are the reason you are here, Ollama and llama.cpp expose them differently.

A recipe for picking num_ctx

1. Read the model's maximum context, its layer count and its key/value head count from /api/show.
2. Work out bytes per token with the formula, then multiply by the context you want.
3. Add the weight size, compare against free RAM, and keep at least 1 GiB back for the rest of the box.
4. Set the value, load the model, then confirm what was applied with ollama ps and prompt_eval_count.
5. Run your real workload while watching free -m, and halve the context if swap starts to move.

Most jobs need less context than people give them. Summarising a long report fits in 16k. A retrieval front end that pastes five document chunks rarely passes 8k. A coding agent that reads whole files is the case that genuinely needs 64k or more, and it is also the case where you should size the machine around the context instead of the other way round. If the server itself is still new, start from a working Ollama install on a VPS and tune the context once models load cleanly.

FAQ

What is the default context length in Ollama?

It depends on the build and on the hardware, so check instead of assuming. Ollama's FAQ documents 4096 tokens, the Modelfile reference documents a num_ctx default of 2048, and the context length page documents a default picked from available VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k above. A CPU-only VPS lands on the small end. ollama ps prints the applied context on builds that carry the column, and prompt_eval_count in an API response proves it on every build.

Why does Ollama ignore the start of my long prompt?

Because the prompt was longer than the context window, so the server cut it before the model saw it, and no error came back. Send the same prompt again with a larger num_ctx and watch prompt_eval_count in the response grow. If that number does not move, something between you and the server is setting num_ctx itself, which is common with chat front ends and agent frameworks.

How much extra RAM does a larger num_ctx need?

Multiply the context length by the cache cost per token, which is 2 * layers * kv_heads * head_dim * bytes_per_value. For Llama 3.1 8B at f16 that is 128 KiB per token, so 32k tokens costs 4 GiB and the full 128k costs 16 GiB on top of the weights. The cache is allocated when the model loads, so a large num_ctx costs that memory even while your prompts stay short.

Does a larger context window make Ollama slower?

Yes, in two ways. Prefill work grows with the square of the prompt length, so a long input delays the first token by more than its length suggests. The larger cache also competes for memory: on a GPU box it pushes layers into system RAM, and on a CPU box it pushes the machine toward swap. A large num_ctx that you never fill still costs the memory, though it does not cost the prefill time.

Can I set num_ctx permanently for one model?

Yes. Write a Modelfile containing FROM llama3.1:8b and PARAMETER num_ctx 16384, then run ollama create llama3.1-16k -f ./Modelfile. Every client that asks for llama3.1-16k gets that context without sending any options. A request carrying its own num_ctx still wins, so this sets a default rather than a ceiling.