SSD Nodes Learn Hosting plans →
How to do am Matt ConnorBy Matt Connor · Updated 2026-10-21

Ollama num_ctx: How to Stop Long Prompts from Cutting

Long prompts fit disappear quietly for Ollama. See why num_ctx defaults differ, how to set am per request or server-wide, and check KV cache RAM before raising am.

Wetin num_ctx dey do, and why your long prompt get cut

Ollama context length na the number of tokens wey loaded model fit hold for memory at once, and num_ctx na the option wey dey set am. Ollama dey choose default wey far below the maximum wey model advertise, so e go cut longer prompt before model even read am. Nothing for the response go tell you say this happen.

Llama 3.1 8B dey listed with 128k context window for Ollama model library. Stock server no go give you that. Ollama own documentation get different defaults for different pages: FAQ talk say na 4096 tokens, Modelfile reference say num_ctx default na 2048, and the context length page say e dey choose default from available VRAM (video RAM): 4k below 24 GiB, 32k from 24 GiB reach 48 GiB, and 256k for 48 GiB and above. Each one dey correct for some build. The useful lesson from this disagreement be say make you read the value from your own running server instead of trusting any page, including this one.

Truncation dey happen quietly because model still dey answer, and the answer still dey read well. E write am from the tail of your input. Summary wey miss the first half of document fit look like say model weak. Most times, na small context window cause am.

Check the Ollama context length wey your server really apply

The check wey work for any build na prompt_eval_count, wey be the number of prompt tokens wey server report say e process. Send prompt wey pass the context capacity, and that number go stop for the limit.

sudo apt update && sudo apt install -y jq
LONG=$(python3 -c "print('the quick brown fox jumps over the lazy dog. ' * 2000)")
jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:4096}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{prompt_eval_count, prompt_eval_duration}'

That prompt get about 18,000 words, so e far pass 4096 tokens. prompt_eval_count go return near 4096 instead of near the real token count, because server drop the remaining part. Run am again with "num_ctx":16384 and the count go increase. If your build return error instead of truncating, na the same finding be that, but the signal go louder.

ollama ps

The CONTEXT column, for builds wey print am, get the context length wey the loaded model dey use now. The PROCESSOR column next to am show where the model dey run. 100% CPU normal for VPS wey no get GPU. A split like 30%/70% CPU/GPU for GPU box mean say the weights plus the cache no fit again inside VRAM, and raised num_ctx na the usual reason.

journalctl -u ollama --no-pager | grep -i n_ctx | tail -n 5

The inference runner print its context size for one line wey contain n_ctx. The exact wording dey change between releases, so treat missing line as a rename, no be proof of anything.

Places four wey you fit set num_ctx

For the request. Send "options": {"num_ctx": 16384} to /api/generate or /api/chat. This one override every other setting, and e apply only to that call. If the value no match wetin the loaded model dey use, the server go reload the model first. You fit see this for load_duration inside the response: e go jump from almost zero to complete seconds. The same wait go show when the model don stay idle reach the point wey dem unload am. So, after you don choose context size, e make sense to keep the model resident with keep_alive.

For the interactive session. Inside ollama run, type /set parameter num_ctx 16384. E go last for that session.

For a Modelfile. This one bake the value inside a named model, so every client go get am without any change for the client side.

FROM llama3.1:8b
PARAMETER num_ctx 16384
ollama create llama3.1-16k -f ./Modelfile
ollama run llama3.1-16k

For the server. OLLAMA_CONTEXT_LENGTH set the default for every request wey no carry its own num_ctx. Under systemd, add a drop-in instead of editing the unit file.

sudo systemctl edit ollama.service
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"
sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama ps

The order of priority matter most when you dey debug another person client. Request wey carry num_ctx go override the server default. So, chat front end or agent wey send small value by itself fit quietly cancel the systemd change wey you make. When you point a coding agent to your Ollama server, check wetin the client dey send before you blame the server.

Why you no fit just set num_ctx to the model maximum

Attention dey make every token look at every token wey come before am. System dey keep the keys and values wey e calculate for earlier tokens, so e no need calculate dem again for every new token. That store na KV cache (key/value cache). System dey allocate am for the whole of num_ctx when model load, not as conversation dey grow. So large context still use memory even when prompt na just one line.

Every layer keep one key vector and one value vector for each key/value head and each token. The arithmetic fit enter one line:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value

The 2 count keys and values separately. Read the other numbers from your own model.

curl -s http://localhost:11434/api/show -d '{"model":"llama3.1:8b"}' |
  jq '.model_info | {ctx: ."llama.context_length", layers: ."llama.block_count", heads: ."llama.attention.head_count", kv_heads: ."llama.attention.head_count_kv", embed: ."llama.embedding_length"}'

Llama 3.1 8B report 32 layers and 8 key/value heads. Head dimension na embed divided by heads, so 4096 / 32 = 128 for here. Some models publish am directly as llama.attention.key_length. Default cache dey hold f16 values, so bytes_per_value na 2. Therefore, 2 32 8 128 2 = 131,072 bytes. That na 128 KiB cache for every single context token. Multiply am by context length, and the cost no be abstract again.

ChartLlama 3.1 8B at f16: KV cache and total RAM by context length, in GiB
Di data wey dey behind dis chart
[
  {
    "label": "4k",
    "kv_cache_gib": 0.5,
    "total_ram_gib": 5.1
  },
  {
    "label": "8k",
    "kv_cache_gib": 1,
    "total_ram_gib": 5.6
  },
  {
    "label": "16k",
    "kv_cache_gib": 2,
    "total_ram_gib": 6.6
  },
  {
    "label": "32k",
    "kv_cache_gib": 4,
    "total_ram_gib": 8.6
  },
  {
    "label": "64k",
    "kv_cache_gib": 8,
    "total_ram_gib": 12.6
  },
  {
    "label": "128k",
    "kv_cache_gib": 16,
    "total_ram_gib": 20.6
  }
]

Those 6 rows na arithmetic from the formula above, not measurements. Total column add the 4.9 GB download wey the Ollama library list for llama3.1:8b for October 2026. That na 4.6 GiB. E no include compute buffers and the server process itself. Treat am as the minimum.

The shape na the main point. For 8k, cache cost 1 GiB, and that small compared with the weights. For the model full 128k, e cost 16 GiB, more than three times the weights, with total near 20.6 GiB. So 4 GB VPS no fit load this model with any useful context. 8 GB VPS comfortable for 8k. 16 GB VPS reach 32k and still get room for the rest of the box. Every one of these limits dey increase with the weights. So if you dey compare larger model with this 8B, the same calculations for Qwen's 27B tag for CPU-only VPS show how small the memory wey weights leave for context between 8 and 64 GB.

Wetin dey happen when KV cache no fit

For CPU-only VPS, the process go simply grow. Watch am while model dey load and while long request dey run.

free -m
ps -eo rss,comm --sort=-rss | head -n 5

RSS (resident set size) dey print for kilobytes. If used swap for free -m start to rise, reduce the context. KV cache wey dey inside swap go make generation pause for seconds for each token, because every new token dey read the whole cache.

If the box memory finish completely, kernel go choose one process to kill, normally the one wey dey hold the most memory.

sudo dmesg | grep -i "killed process"

Line wey read Out of memory: Killed process 1234 (ollama) mean say Ollama na the process wey kernel choose, no be necessarily the one wey use up the memory. For box where Ollama na the only large process, context wey too big to fit na the usual cause. Check wetin else dey run before you reduce num_ctx. Ollama often dey refuse before e reach that point, then the request go fail with message wey name the memory e want compared with the memory wey free.

For GPU box, the failure no loud. Layers go spill enter system RAM, ollama ps go show the CPU and GPU split, and throughput go drop sharply. How sharp the drop go be depend on your hardware, so measure tokens per second for your own box for each context setting instead of trusting figure from another person machine.

Prefill time dey grow faster than the prompt

Prefill na the work wey happen for your input before the first output token show. Each prompt token dey attend to every token before am, so the total work dey grow with the square of the input length. If you double the prompt, the wait for the first token go more than double.

The response carry the measurement, so you no need just trust am.

jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:16384}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{tokens: .prompt_eval_count, prefill_seconds: (.prompt_eval_duration/1000000000)}'

Run am with short prompt, then run am again with long one. For each run, divide tokens by seconds. For CPU-only VPS, prefill usually na the slowest part of long-context request. So, tokens per second wey you measure with short prompt no go predict long prompt performance.

If prefill take pass the timeout wey dey in front of am, long prompt fit return context deadline exceeded instead of answer. Na this one usually cause the problem. Before you reduce context size, find out which layer stop waiting and give up.

Concurrency na where this matter pass. Every request wey service dey handle need its own cache, so the memory for the chart above na per request, no be per server. One long request fit hold the whole box while short ones queue behind am. Set OLLAMA_NUM_PARALLEL deliberately, and read how many concurrent users one self-hosted LLM fit serve before you increase both numbers together.

Context wey smaller cache fit buy back

bytes_per_value for the formula na setting wey you fit control. Ollama FAQ document say OLLAMA_KV_CACHE_TYPE, with f16 as the default, q8_0 dey use about half the memory of f16, and q4_0 dey use about one-quarter. The exact figure for q8_0 small pass half: llama.cpp dey store each block of 32 values as 32 one-byte values plus one two-byte scale, so na 34 bytes, while f16 need 64. If you move go q8_0, the 32k row go use about 2.1 GiB instead of 4 GiB. Quantising the weights free memory from the other side of the same budget, and the GLM tag wey really fit VPS dey show the trade-off one quantisation at a time. The same FAQ talk say cache only dey quantised when Flash Attention dey enabled, and e document OLLAMA_FLASH_ATTENTION=1 to force am on.

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Confirm am instead of assuming: restart the service, load the model with the same num_ctx as before, then compare RSS. Support depend on the model and backend, so if setting no change anything, your combination no dey covered. Cache type na global option, so every model for the server go run with am. For quality, the FAQ talk say q8_0 usually no get noticeable effect, while q4_0 lose more precision, especially when context size big. So test q4_0 with your own prompts before you depend on am. If na these settings bring you here, Ollama and llama.cpp expose dem differently.

Recipe for how to choose num_ctx

  1. Read the model maximum context, layer count, and key/value head count from /api/show.
  2. Calculate bytes per token with the formula, then multiply am with the context wey you want.
  3. Add the weight size, compare am with free RAM, and leave at least 1 GiB for the rest of the server.
  4. Set the value, load the model, then confirm wetin apply with ollama ps and prompt_eval_count.
  5. Run your real workload while you dey monitor free -m, and cut the context by half if swap start to move.

Most jobs no need as much context as people dey give dem. You fit summarise one long report with 16k. Retrieval front end wey dey paste five document chunks rarely pass 8k. Coding agent wey dey read complete files na the case wey really need 64k or more. Na also this case make you size the machine around the context, instead of doing am the other way round. If the server still new, start from working Ollama installation for VPS and tune the context after models dey load cleanly.

FAQ

Wetin be the default context length for Ollama?

E depend on the build and the hardware, so check am instead of assuming. Ollama FAQ document 4096 tokens, the Modelfile reference document a num_ctx default of 2048, and the context length page document a default wey e pick from available VRAM: 4k below 24 GiB, 32k from 24 GiB reach 48 GiB, and 256k for 48 GiB and above. CPU-only VPS go dey for the small end. ollama ps go print the applied context for builds wey get that column, and prompt_eval_count for API response go prove am for every build.

Why Ollama dey ignore the beginning of my long prompt?

Na because the prompt long pass the context window, so the server cut am before the model see am, and no error return. Send the same prompt again with bigger num_ctx and monitor prompt_eval_count as e grow for the response. If that number no move, something between you and the server dey set num_ctx by itself. This one common with chat front ends and agent frameworks.

How much extra RAM larger num_ctx need?

Multiply the context length by the cache cost per token, wey be 2 * layers * kv_heads * head_dim * bytes_per_value. For Llama 3.1 8B at f16, na 128 KiB per token, so 32k tokens cost 4 GiB and the full 128k cost 16 GiB on top the weights. The cache dey allocate when the model load, so large num_ctx go use that memory even when your prompts short.

Larger context window go make Ollama slower?

Yes, for two ways. Prefill work dey grow with the square of the prompt length, so long input go delay the first token pass wetin the length alone suggest. The bigger cache still dey compete for memory: for GPU box, e go push layers enter system RAM, and for CPU box, e go push the machine toward swap. Large num_ctx wey you never fill still dey cost the memory, though e no cost the prefill time.

I fit set num_ctx permanently for one model?

Yes. Write Modelfile wey contain FROM llama3.1:8b and PARAMETER num_ctx 16384, then run ollama create llama3.1-16k -f ./Modelfile. Every client wey ask for llama3.1-16k go get that context without sending any options. Request wey carry e own num_ctx still get priority, so this one set default, no be ceiling.

Dis article help you?