Ollama num_ctx: How to Stop Long Prompts Cutting

Wetin num_ctx dey do, and why your long prompt get cut

Ollama context length na the number of tokens wey loaded model fit keep for memory at once, and num_ctx na the option wey dey set am. Ollama dey choose default wey far below the maximum wey model advertise, so e go cut longer prompt before model even read am. Nothing for the response go tell you say e happen.

Llama 3.1 8B dey listed with 128k context window for Ollama model library. Stock server no go give you that. Ollama own documentation get different defaults for different pages: FAQ talk say 4096 tokens, Modelfile reference talk say num_ctx default na 2048, and the context length page talk say default dey come from available VRAM (video RAM): 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k above that. Each one correct for some build. The useful lesson from that disagreement be say make you read the value from your own running server instead of trusting any page, including this one.

Truncation dey happen quietly because model still dey answer, and the answer still dey read well. E write am from the tail end of your input. Summary wey miss the first half of document fit look like say model weak. Most times, na small context window cause am.

Check the Ollama context length wey your server really apply

The check wey dey work for any build na prompt_eval_count, wey be the number of prompt tokens wey server report say e process. Send more than the context fit hold, and that number go stop for the limit.

sudo apt update && sudo apt install -y jq
LONG=$(python3 -c "print('the quick brown fox jumps over the lazy dog. ' * 2000)")
jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:4096}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{prompt_eval_count, prompt_eval_duration}'

That prompt get about 18,000 words, so e pass 4096 tokens by far. prompt_eval_count go return close to 4096 instead of the real token count, because server drop the remaining part. Run am again with "num_ctx":16384 and the count go rise. If your build return error instead of truncating, na the same finding be that, but the signal loud pass.

ollama ps

The CONTEXT column, for builds wey print am, hold the context length wey the loaded model dey use now. The PROCESSOR column beside am show where the model dey run. 100% CPU normal for VPS wey no get GPU. Split like 30%/70% CPU/GPU for GPU machine mean say the weights plus the cache no fit inside VRAM again, and raised num_ctx na the usual reason.

journalctl -u ollama --no-pager | grep -i n_ctx | tail -n 5

The inference runner print its context size for a line wey contain n_ctx. The exact wording dey change between releases, so if the line no show, treat am as rename, not as proof of anything.

Places wey you fit set num_ctx

For the request. Send "options": {"num_ctx": 16384} to /api/generate or /api/chat. Dis one get priority pass every other setting, and e apply only to dat one call. If di value no match wetin di loaded model dey use, di server go reload di model first. You fit see am for load_duration for di response: e go jump from almost zero reach full seconds.

For di interactive session. Inside ollama run, type /set parameter num_ctx 16384. E go last for dat session.

For Modelfile. Dis one go bake di value inside a named model, so every client go get am without any change for di client side.

FROM llama3.1:8b
PARAMETER num_ctx 16384

ollama create llama3.1-16k -f ./Modelfile
ollama run llama3.1-16k

For di server. OLLAMA_CONTEXT_LENGTH go set di default for every request wey no carry im own num_ctx. Under systemd, add drop-in instead of editing di unit file.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"

sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama ps

Precedence dey important pass when you dey debug another person client. Request wey carry num_ctx go override di server default, so chat front end or agent wey send small value by itself fit quietly cancel di systemd change wey you make. When you point coding agent go your Ollama server, check wetin di client dey send before you blame di server.

Why you no fit just set num_ctx to the model maximum

Attention dey make every token look at every token wey come before am. The keys and values wey e calculate for earlier tokens dey kept so e no need calculate dem again for every new token. That storage na the KV cache (key/value cache). E dey allocate for the whole of num_ctx when model load, no be as conversation dey grow. So large context go use the memory even for one-line prompt.

DigitalOcean's inference cost tutorial state the calculation for one line:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value

The 2 count keys and values separately. Read the other numbers from your own model.

curl -s http://localhost:11434/api/show -d '{"model":"llama3.1:8b"}' |
  jq '.model_info | {ctx: ."llama.context_length", layers: ."llama.block_count", heads: ."llama.attention.head_count", kv_heads: ."llama.attention.head_count_kv", embed: ."llama.embedding_length"}'

Llama 3.1 8B report 32 layers and 8 key/value heads. Head dimension na embed divided by heads, so 4096 / 32 = 128 for here. Some models publish am directly as llama.attention.key_length. Default cache dey hold f16 values, so bytes_per_value na 2. Therefore, 2 32 8 128 2 = 131,072 bytes. That mean cache for every single context token na 128 KiB. Multiply am by context length, and the cost no dey abstract again.

[
  {
    "label": "4k",
    "kv_cache_gib": 0.5,
    "total_ram_gib": 5.1
  },
  {
    "label": "8k",
    "kv_cache_gib": 1,
    "total_ram_gib": 5.6
  },
  {
    "label": "16k",
    "kv_cache_gib": 2,
    "total_ram_gib": 6.6
  },
  {
    "label": "32k",
    "kv_cache_gib": 4,
    "total_ram_gib": 8.6
  },
  {
    "label": "64k",
    "kv_cache_gib": 8,
    "total_ram_gib": 12.6
  },
  {
    "label": "128k",
    "kv_cache_gib": 16,
    "total_ram_gib": 20.6
  }
]

Those 6 rows na calculation from the formula above, no be measurements. The total column add the 4.9 GB download wey the Ollama library list for llama3.1:8b for August 2026. That na 4.6 GiB. E no include compute buffers or the server process itself. Treat am as the minimum.

The shape na the main point. For 8k, cache cost 1 GiB, and that small amount no matter beside the weights. For the model full 128k, e cost 16 GiB, more than three times the weights, with total near 20.6 GiB. So 4 GB VPS no fit load this model at any useful context. 8 GB VPS comfortable for 8k. 16 GB VPS fit reach 32k and still get space for the rest of the box. Every one of these limits dey increase as the weights increase. So if you dey compare bigger model with this 8B, the same calculations for Qwen 27B tag for CPU-only VPS show how little space the weights leave for context between 8 and 64 GB.

Wetin dey happen when KV cache no fit

For CPU-only VPS, the process go simply grow. Monitor am while the model dey load and while long request dey run.

free -m
ps -eo rss,comm --sort=-rss | head -n 5

RSS (resident set size) dey print for kilobytes. If used swap for free -m start to increase, reduce the context. KV cache wey dey inside swap go make generation pause for seconds for each token, because every new token dey read the whole cache.

If the box run out of memory completely, kernel go pick the biggest process and kill am.

sudo dmesg | grep -i "killed process"

Line wey read Out of memory: Killed process 1234 (ollama) mean say the context wey you request no fit. Ollama often dey refuse before e reach that point, and the request go fail with message wey name the memory e need against the memory wey dey free.

For GPU box, the failure no too obvious. Some layers go spill into system RAM, ollama ps go show the CPU and GPU split, and throughput go drop sharply. How sharp the drop go be depend on your hardware, so measure tokens per second for your own box for each context setting instead of trusting figure from another person machine.

Prefill time dey grow faster than the prompt

Prefill na the work wey happen for your input before the first output token show. Each prompt token dey attend to every token before am, so the total work dey grow with the square of the input length. If you double the prompt, the wait for the first token go pass double.

The response carry the measurement, so you no need just trust am.

jq -n --arg p "$LONG" '{model:"llama3.1:8b", prompt:$p, stream:false, options:{num_ctx:16384}}' |
  curl -s http://localhost:11434/api/generate -d @- |
  jq '{tokens: .prompt_eval_count, prefill_seconds: (.prompt_eval_duration/1000000000)}'

Run am with short prompt and run am again with long one, then divide tokens by seconds for each case. For CPU-only VPS, prefill usually na the slowest part of long-context request, and tokens per second figure wey you take from short prompt no go predict am.

Concurrency na where this matter pass. Every request wey server dey handle need im own cache, so the memory for the chart above na per request, no be per server. One long request fit hold the whole box while short ones queue behind am. Set OLLAMA_NUM_PARALLEL deliberately, and read how many concurrent users one self-hosted LLM fit serve before you increase both numbers together.

Buy context back with smaller cache

bytes_per_value for the formula na setting wey you fit control. Ollama FAQ document OLLAMA_KV_CACHE_TYPE, with f16 as the default for 2 bytes, plus q8_0 for 1 byte and q4_0 below that. If you move to q8_0, the cache go reduce by half, so the 32k row go cost 2 GiB instead of 4 GiB. The same FAQ document OLLAMA_FLASH_ATTENTION=1, wey some builds need before quantised cache go take effect.

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

No assume; confirm am: restart the service, load the model with the same num_ctx wey you use before, then compare RSS. Support depend on the model and backend, so if the setting no change anything, your combination no dey covered. The documentation list these options but e no promise any quality result, so test q4_0 with your own prompts before you depend on am. If na these knobs bring you here, Ollama and llama.cpp expose dem differently.

Recipe for how to choose num_ctx

1. Read the model maximum context, layer count, and key/value head count from /api/show.
2. Calculate bytes per token with the formula, then multiply am with the context wey you want.
3. Add the weight size, compare am with free RAM, and leave at least 1 GiB for the rest of the machine.
4. Set the value, load the model, then confirm wetin apply with ollama ps and prompt_eval_count.
5. Run your real workload while you dey monitor free -m, and cut the context by half if swap start to move.

Most jobs no need as much context as people dey give dem. To summarise long report fit work with 16k. Retrieval front end wey paste five document chunks rarely pass 8k. Coding agent wey read complete files na the case wey really need 64k or more. Na also this case make you suppose size the machine around the context, instead of doing am the other way round. If the server still new, start from a working Ollama installation for VPS and tune the context after models dey load cleanly.

FAQ

Wetin be the default context length for Ollama?

E depend on the build and the hardware, so check am instead of assuming. Ollama FAQ document 4096 tokens, the Modelfile reference document a num_ctx default of 2048, and the context length page document a default wey e pick from available VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k above. CPU-only VPS go dey for the small end. ollama ps dey print the applied context for builds wey carry the column, and prompt_eval_count for an API response prove am for every build.

Why Ollama dey ignore the beginning of my long prompt?

Na because the prompt long pass the context window, so the server cut am before the model see am, and no error return. Send the same prompt again with bigger num_ctx and monitor prompt_eval_count as e dey grow for the response. If that number no move, something between you and the server dey set num_ctx by itself. This one common with chat front ends and agent frameworks.

How much extra RAM larger num_ctx need?

Multiply the context length by the cache cost per token, wey be 2 * layers * kv_heads * head_dim * bytes_per_value. For Llama 3.1 8B at f16, na 128 KiB per token. So 32k tokens cost 4 GiB, and the full 128k cost 16 GiB on top of the weights. The cache dey allocate when the model load, so large num_ctx go use that memory even when your prompts short.

Larger context window dey make Ollama slower?

Yes, for two ways. Prefill work dey grow with the square of the prompt length, so long input fit delay the first token pass wetin the length alone suggest. The larger cache also dey compete for memory: for GPU box, e fit push layers go system RAM; for CPU box, e fit push the machine toward swap. Large num_ctx wey you never fill still dey use the memory, though e no use the prefill time.

I fit set num_ctx permanently for one model?

Yes. Write a Modelfile wey contain FROM llama3.1:8b and PARAMETER num_ctx 16384, then run ollama create llama3.1-16k -f ./Modelfile. Every client wey ask for llama3.1-16k go get that context without sending any options. Request wey carry im own num_ctx still win, so this one set a default, no be a ceiling.