Reasoning effort settings on a local LLM

What reasoning effort changes on a local LLM

Reasoning effort is a setting that tells a model how long to think before it answers. It changes the length of the reasoning segment and nothing else. The weights on disk are identical at every level, the quantisation is identical, and the answer comes out of the same forward pass. What moves is how many tokens the model spends on its own scratchpad first.

That distinction matters because of where those tokens land. On a hosted API, reasoning tokens show up on an invoice. On a VPS you own, they are paid in generation time on your own CPU or GPU, and in space inside the context window. A model left at its highest effort can spend most of its output on reasoning before the first word of the answer appears, which on self-hosted hardware is the difference between a two second reply and a two minute one.

Where the level lives: the chat template, not the weights

A thinking model is trained to emit a reasoning segment, usually wrapped in <think> and </think> tags, before its final answer. The effort level is an instruction that the model's chat template writes into the prompt. That template is a Jinja file shipped with the model. It reads a variable such as reasoning_effort and renders a different system-level line for each value, and the model was trained to shorten or lengthen its scratchpad in response to that line.

Two things follow. The level names belong to the model and not to your runtime, so a name from one model's card can mean nothing to another. And if anything in the chain swaps the model's chat template for a generic one, the variable is never rendered and the setting silently does nothing.

Checked on 2026-08-20, the Qwen3.8-27B model card documents three effort levels: low, medium and xhigh, with xhigh as the default. There is no high. Thinking itself is switched with enable_thinking, which is on by default, and the card also documents preserve_thinking, on by default, which keeps reasoning from earlier turns in the conversation history. gpt-oss uses low, medium and high instead. Many other families accept a boolean and nothing more. Read the card for the exact version you pulled, because these names are not a standard. Getting a 27B model running on a VPS comes first. This page is about what to set once it answers.

Why high effort costs more on a VPS

Output tokens. Reasoning tokens are generated tokens. They go through the same decode loop as the answer, at the same tokens per second your hardware manages. Suppose a task produces 200 tokens of answer and 4,000 tokens of reasoning. You generated 4,200 tokens and the reader saw 200 of them. Your decode rate is set by memory bandwidth and by the quantisation you picked, so the only lever left is the token count itself.

Wall clock. A person waits for the first token of the answer, because everything before that is a blank screen or a collapsed spinner. Reasoning is emitted first, so the wait is roughly the reasoning token count divided by your decode rate, plus prompt processing. Double the reasoning length and you double that wait.

Context. Reasoning tokens occupy the context window like any other token. With preserve_thinking on, the scratchpad from turn one is still in the prompt at turn five, so prompt processing gets slower every turn while the window fills from both ends. Raising num_ctx to hold it costs KV cache memory, which on a VPS without a GPU is system RAM you may not have spare.

When to raise the level, and when to leave it low

Raise it for work where a wrong intermediate step poisons the result: multi-step arithmetic and unit conversion, planning an edit across several files, code that has to compile, and constraint problems where one answer must satisfy several conditions at once. In these the scratchpad is doing real work, and a longer one is a cheap way to catch an error the model would otherwise commit to.

Leave it low when the answer is already in the input and the job is to move it. Extraction, classification, tagging, translation, rewriting, summarising and formatting all fall here. The reasoning segment mostly restates the task, and it gives the model room to argue itself out of a correct first instinct.

Leave it low for anything interactive as well. In a chat box or an editor you are in the loop, so a fast answer you can correct beats a slow one you have to wait for. That is the real trade-off behind pointing a coding agent at a local model: an agent makes many small calls, and the reasoning tax is charged on every one of them.

How to set the level in llama.cpp

llama.cpp writes the variable into the template directly, which makes it the runtime where you can be certain the level arrived. Point -m at the GGUF you already have.

llama-server -m ./qwen3.8-27b-Q4_K_M.gguf \
  --jinja \
  --reasoning-effort medium \
  --reasoning-format deepseek \
  -c 32768 \
  --host 127.0.0.1 --port 8080

--jinja uses the model's own chat template and is enabled by default in current builds. --reasoning-effort accepts default, minimal, low, medium, high, xhigh or max, where default means leave the template's own default alone. That list is llama.cpp's vocabulary and not the model's, so pass only a name the card lists: a level the template does not define can raise a template error at request time. --reasoning-format deepseek moves the reasoning out of message.content and into message.reasoning_content, which is what makes the split measurable in the next section.

To switch thinking off rather than shorten it, set the template variable yourself:

llama-server -m ./qwen3.8-27b-Q4_K_M.gguf --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

--reasoning-budget is a different mechanism. It caps the reasoning segment in tokens, with 0 ending it immediately and -1 leaving it unrestricted, rather than asking the model to plan a shorter one. Both flags are server wide. llama-server does not take reasoning_effort as a per-request field, so serving two effort levels at the same time means two processes on two ports.

vLLM exposes the same variable per request, inside the OpenAI-compatible body:

{"model": "Qwen/Qwen3.8-27B",
 "messages": [{"role": "user", "content": "Summarise this changelog in two lines."}],
 "chat_template_kwargs": {"reasoning_effort": "medium"}}

How to set the level in Ollama

Ollama has its own field, think, on /api/chat and /api/generate. It accepts true, false, or one of low, medium, high and max, where max asks for the highest level the model offers. Thinking is on by default for models that support it.

ollama run qwen3.8:27b --think=low "Draft a one line commit message for a README typo fix"

{"model": "qwen3.8:27b",
 "messages": [{"role": "user", "content": "Which HTTP status code means the request body was too large?"}],
 "think": "low",
 "stream": false}

The reasoning comes back in message.thinking and the answer in message.content, already split for you. Inside an interactive ollama run session, /set think and /set nothink toggle it without restarting.

Now notice the mismatch. Ollama's vocabulary is low, medium, high and max. Qwen3.8's template defines low, medium and xhigh. Something has to map one onto the other, and an Ollama model carries a template packaged inside its tag rather than the Jinja file from the original repository, so whether your level reaches the model depends on that packaged template. Do not assume it worked. Measuring it takes about a minute.

How to measure whether the level actually landed

Send the same prompt at more than one level with temperature at 0, then compare the token counts. Here jq builds the body so you do not have to escape quotes by hand.

for level in low medium max; do
  body=$(jq -n --arg lvl "$level" '{
    model: "qwen3.8:27b",
    messages: [{role: "user", content: "A pump fills a 4500 litre tank in 25 minutes. A second pump is 40 percent slower. How long do both together take? Answer in minutes."}],
    think: $lvl,
    stream: false,
    options: {temperature: 0, num_ctx: 8192}
  }')
  echo "== $level"
  curl -s http://localhost:11434/api/chat -d "$body" | jq '{
    thinking_chars: (.message.thinking // "" | length),
    answer_chars: (.message.content | length),
    eval_count: .eval_count,
    seconds: (.total_duration / 1e9),
    tok_per_sec: (.eval_count / (.eval_duration / 1e9))
  }'
done

eval_count is every token generated, reasoning included, so the gap between two levels is almost all reasoning. thinking_chars gives you the split directly. Two things should be true: the numbers move between levels, and the answer stays right at the lower one. If eval_count sits within noise across all three runs, the level is being ignored, and the fix is a runtime that passes it rather than a different level name.

Total time is only half the story, so measure the gap to the first answer token by streaming and stopping at the first non-empty content chunk. This one needs jq and bc.

start=$(date +%s.%N)
curl -sN http://localhost:11434/api/chat -d '{
  "model": "qwen3.8:27b",
  "messages": [{"role": "user", "content": "Explain what a reverse proxy does, in three sentences."}],
  "think": "low",
  "stream": true
}' |
while IFS= read -r line; do
  if [ -n "$(printf '%s' "$line" | jq -r '.message.content // ""')" ]; then
    echo "first answer token after $(echo "$(date +%s.%N) - $start" | bc)s"
    break
  fi
done

Run it at low and again at max. The difference is the wait you are buying. On llama.cpp the same numbers come back inside the response, with no shell arithmetic needed:

curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "local", "temperature": 0,
       "messages": [{"role": "user", "content": "A pump fills a 4500 litre tank in 25 minutes. A second pump is 40 percent slower. How long do both together take?"}]}' | jq '{
  reasoning_chars: (.choices[0].message.reasoning_content // "" | length),
  answer_chars: (.choices[0].message.content | length),
  predicted_n: .timings.predicted_n,
  tok_per_sec: .timings.predicted_per_second
}'

Do this on your own box. A published effort comparison was measured on hardware that is not yours, and your decode rate is the term that turns a token count into seconds. Measuring tokens per second on your own server gives you that term: reasoning tokens divided by your decode rate is the wait you just added.

What goes wrong

The answer is cut off, or content is empty while thinking is full. The generation limit was eaten by reasoning. Ollama's num_predict caps the whole generation, reasoning included, and reasoning comes first, so a cap of 512 tokens at high effort can end the response before the answer starts. Ollama reports "done_reason": "length" on that response. Raise the cap or lower the effort. How num_predict counts tokens covers the interaction in detail.

The level changes nothing. Token counts are identical at every level. Either the runtime is not passing the variable, or the template does not read it. Check the template your runtime is actually using instead of the one in the original repository. llama.cpp with --jinja and --chat-template-kwargs writes the variable in by hand, so it makes a good control: if the level works there and nowhere else, the model is fine and the other runtime is dropping it.

A level name is rejected. A template error at request time, or a failure on the first message with an otherwise healthy server, usually means you passed a level the template does not define, such as high to a model whose card lists only low, medium and xhigh.

Multi-turn chats slow down every turn. Old reasoning is being kept in the history. Set preserve_thinking to false if the model supports it, or strip the thinking field from the messages you send back. Otherwise prompt processing grows on each turn while the answers stay the same length.

Quality drops at low effort on a task you thought was simple. Some extraction is not extraction. If the input needs a unit conversion or a rule applied in order, it is a reasoning task with a short output. Raise the level for that one call rather than for the whole server.

Running two levels at once

llama.cpp fixes the level at start-up, so a box serving both an editor and a nightly batch job wants two processes on two ports, each with its own --reasoning-effort. Two processes also means two copies of the weights in memory, unless you separate the jobs in time instead. On one VPS the cheaper arrangement is usually a low effort server for anything a person is waiting on, plus a scheduled higher effort run for work nobody is watching. What happens when several users share one local model applies here too: reasoning tokens are decode work, so raising the effort cuts your effective concurrency by roughly the same factor it raises the token count.

FAQ

Which reasoning effort level should I use by default?

Start at the lowest level the model offers and raise it only for tasks you have watched fail. Several thinking models ship with a high default, and Qwen3.8-27B defaults to xhigh, its top level, as of August 2026. That default is chosen to look good on benchmark tables, and a benchmark table does not charge for time. On your own hardware you pay in seconds, so make the higher level something you opt into per task rather than the setting every request inherits.

Do reasoning tokens count against my context window?

Yes. They are ordinary tokens in the output and they sit in the context window along with everything else. Whether they stay there on the next turn depends on the runtime and the model. Qwen3.8's card documents preserve_thinking, on by default, which keeps earlier reasoning in the history, so a long conversation carries every scratchpad it has produced. Set it to false, or drop the thinking field from the messages you replay, and prompt processing stops growing.

Why does changing the thinking level make no difference to my token counts?

The setting is not reaching the chat template. The level is a template variable, so it works only if the runtime passes it and the packaged template reads it. Some runtimes ship their own template with a model instead of the Jinja file from the original repository, and the variable is then dropped with no error printed anywhere. Prove it by sending the same prompt at the lowest and the highest level with temperature at 0 and comparing eval_count. If the counts match within noise, the level is being ignored.

Does lower reasoning effort make the model less accurate?

It depends on the task, and this is worth measuring rather than assuming. Where the answer is already present in the input, such as extraction or rewriting, a shorter scratchpad usually changes nothing. Where an intermediate step has to be right before the final one can be, such as multi-step arithmetic or code that must compile, accuracy does fall with a shorter scratchpad. Build a set of twenty prompts from your real workload, run them at two levels with temperature at 0, and count the wrong answers. That number is specific to your workload, and no published table can give it to you.