Run Qwen 3.8 27B on a VPS With Ollama

Can you run Qwen 3.8 27B on a VPS with no GPU?

To run Qwen 3.8 27B on a VPS you need the qwen3.8:27b tag, and as of 21 October 2026 the Ollama library carries it as a locally pullable tag: 27.3 billion parameters, Q4_K_M quantisation, Apache 2.0 licence, a 256K context window, and an 18 GB download that includes a vision projector of about 0.9 GB. Ollama added Qwen 3.8 27B support in v0.32.12, released 14 August 2026, so an older install needs an upgrade first. As of 29 October 2026 the latest stable release is v0.40.13, published 28 October 2026.

The short answer is yes on a 32 GB or larger VPS, and slowly. A 27B dense model at Q4 needs about 17 GB of RAM for weights alone, before a single token of context is stored. That rules out 8 GB and 16 GB plans completely. On a common two-channel DDR4 VPS the ceiling is roughly 3 tokens per second, which is slower than most people read.

The earlier 27B tags are still in the library: qwen3.6:27b and qwen3.5:27b are both Q4_K_M builds of similar size. Qwen3.8-27B keeps the architecture of Qwen3.6-27B, and the official config files for the two give the same layer layout, so the memory arithmetic below applies to either. Check the live list before you copy any command, at the Ollama qwen3.8 tag page. The arithmetic here depends on parameter count, bits per weight and attention layout rather than on the version number.

Which Ollama tag to pull, and how to check

Pulling a tag that does not exist gives a clear error, so this is quick to settle on the box itself. A tag that does exist can still be unrunnable locally, which is what trips people up with GLM 5.2, listed in the library but served only from Ollama's cloud.

ollama --version
ollama pull qwen3.8:27b
ollama show qwen3.8:27b

ollama show prints the architecture, the parameter count, the context length and the quantisation for the tag you actually have. If the parameter line reads 27.3B and the quantisation line reads Q4_K_M, you have the build this guide is written against. The library also carries qwen3.8:27b-q8_0 and qwen3.8:27b-bf16 for the same weights at higher precision, 27b-mtp variants, and 27b-nvfp4, 27b-mxfp8, 27b-mlx and 27b-mlx-bf16 tags that the tags page labels MLX, a format aimed at Apple Silicon rather than a Linux VPS. Unlike qwen3.6, the qwen3.8 entry has no 35b-a3b tags. Those MoE (mixture of experts) models still sit under qwen3.6 and behave very differently on CPU. More on those below.

Parameter count times bytes per weight

[
  {
    "label": "Q4_K_M",
    "size_gb": 17,
    "bits_per_weight": 4.98,
    "notes": "published tag qwen3.8:27b, language model file"
  },
  {
    "label": "Q5_K_M",
    "size_gb": 19.5,
    "bits_per_weight": 5.7,
    "notes": "computed, no official library tag exists"
  },
  {
    "label": "NVFP4",
    "size_gb": 18,
    "bits_per_weight": 5.27,
    "notes": "published tag 27b-nvfp4, MLX build, one file including the vision encoder"
  },
  {
    "label": "Q8_0",
    "size_gb": 29,
    "bits_per_weight": 8.5,
    "notes": "published tag 27b-q8_0, language model file"
  },
  {
    "label": "BF16",
    "size_gb": 55,
    "bits_per_weight": 16.1,
    "notes": "published tag 27b-bf16, language model file"
  }
]

The formula is one line. Bytes of weights = parameters * bits per weight / 8. At a clean 4 bits, 27.3 billion parameters would be 13.7 GB. The language model file in the shipped Q4_K_M tag is 17 GB, which works out at 4.98 bits per weight in practice.

That gap is not an error. K-quant formats do not store every tensor at the nominal width. The tensors that lose the most quality under compression get more bits. Ollama's tensor listing for the shipped qwen3.8:27b file shows these at Q6_K: the output layer, about half of the feed-forward down projections, the attention V projections in about half of the full-attention layers, and about half of the fused QKV projections in the linear-attention layers. The small normalisation tensors stay at full 32-bit precision. The token embedding, the attention Q and K projections and the feed-forward gate and up projections stay at Q4_K like most of the model. The name on the format is an average, and the average lands near 5. The same effect shows at the other end of the scale: 55 GB for BF16 is 16.1 bits per weight rather than a flat 16. There the token embedding is BF16 like everything else, and the small excess comes from the file's metadata, the normalisation tensors kept at 32-bit, and the listed size being rounded to the nearest gigabyte.

Budget for the whole tag, not just the language model file. Every GGUF tag of qwen3.8:27b also ships a vision projector of about 0.9 GB, which is why the Q4_K_M tag is listed at 18 GB.

Q5_K_M has no published tag in the official qwen3.8 library entry, so the 19.5 GB row is computed at the usual 5.7 bits per weight for that format rather than measured. Q8_0 nearly doubles Q4 to 29 GB. On a CPU-only box that doubling costs twice the memory traffic per token, so it also roughly halves your tokens per second. Q4_K_M is the right default here for that reason alone. If you want the quality side of that decision rather than the memory side, a closer comparison of Q4, Q8 and fp16 shows where the output actually starts to degrade.

What the KV cache costs as the context grows

The weights are a fixed cost. The KV cache (key and value cache, the attention state the model keeps for every token it has already seen) grows in a straight line with context length, and it is where most people actually run out of RAM.

[
  {
    "label": "4k tokens",
    "kv_f16_gb": 0.25,
    "kv_q8_gb": 0.13
  },
  {
    "label": "8k tokens",
    "kv_f16_gb": 0.5,
    "kv_q8_gb": 0.25
  },
  {
    "label": "16k tokens",
    "kv_f16_gb": 1,
    "kv_q8_gb": 0.5
  },
  {
    "label": "32k tokens",
    "kv_f16_gb": 2,
    "kv_q8_gb": 1
  },
  {
    "label": "64k tokens",
    "kv_f16_gb": 4,
    "kv_q8_gb": 2
  },
  {
    "label": "128k tokens",
    "kv_f16_gb": 8,
    "kv_q8_gb": 4
  }
]

Those figures come from the model's own config file. Qwen3.8-27B, like Qwen3.6-27B, is a hybrid-attention model. Only 16 of its 64 layers use full attention and keep a cache that grows with every token, and those layers have 4 key/value heads under GQA (grouped-query attention) and a head dimension of 256. The other 48 layers use linear attention (Gated DeltaNet) and hold a fixed-size state instead, which by my arithmetic is on the order of 150 MB per conversation slot however long the context gets. That comes to 64 KiB per token at f16, so 2 GB at 32k tokens and 8 GB at 128k. Do not take my arithmetic over your own box. Load the model and read the SIZE column of ollama ps, which reports weights plus cache plus overhead as a single figure.

The 256K context on the model card is within reach on this architecture, but it is not free. Filling it at f16 costs about 16 GB of cache on top of the weights, on a machine that already spent 17 GB on weights. Ollama does not hand you the full window by default. Its context-length documentation sets the default by VRAM: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. A CPU-only VPS has no VRAM at all, so it lands on the 4k tier, and you raise it deliberately with OLLAMA_CONTEXT_LENGTH. That server-wide variable is not the only lever, and setting num_ctx on the individual request lets you keep a cheap default for everything else while one long job gets the bigger window. Raise it in steps and check ollama ps after each change.

Two settings keep the cache down. OLLAMA_KV_CACHE_TYPE=q8_0 stores the cache at 8 bits instead of 16, taking 32k tokens from 2 GB down to 1 GB. It needs flash attention, so set OLLAMA_FLASH_ATTENTION=1 as well, and confirm the drop in ollama ps instead of assuming it applied. If you have not touched the cache type before, a walkthrough of q8_0 against f16 for Ollama's KV cache covers setting both variables and reading the before and after figures. OLLAMA_NUM_PARALLEL=1 matters just as much. Ollama can serve several requests at once, and each slot gets its own slice of context, so raising parallelism quietly multiplies the cache you budgeted for. The current default is already 1, and the override below pins it there. If more than one person will use this box, that multiplication is where the trouble starts, and the number of concurrent users a self-hosted model can serve is decided by cache slots and queue depth long before it is decided by core count.

What fits in 8, 16, 32 and 64 GB of RAM

[
  {
    "label": "8 GB",
    "q4_max_ctx_ktok": 0,
    "q8_max_ctx_ktok": 0,
    "notes": "Weights alone exceed the box. Use a 4b or 8b model."
  },
  {
    "label": "16 GB",
    "q4_max_ctx_ktok": 0,
    "q8_max_ctx_ktok": 0,
    "notes": "17 GB of weights does not fit in 16 GB of RAM."
  },
  {
    "label": "32 GB",
    "q4_max_ctx_ktok": 128,
    "q8_max_ctx_ktok": null,
    "notes": "Q4 fits with room for long context. Q8 weights fit, but usable context needs a measured total memory budget."
  },
  {
    "label": "64 GB",
    "q4_max_ctx_ktok": 256,
    "q8_max_ctx_ktok": 256,
    "notes": "Both fit with the full 256K window."
  }
]

Read the two numbers as thousands of tokens of context that fit alongside the weights and the vision projector, at f16 cache, on a headless Linux VPS with about 1.5 GB left for the operating system and a small margin on top. A zero means the weights themselves do not fit, so nothing fits; null means the weights fit but no usable context figure is confirmed for that tier.

8 GB and 16 GB are not close calls. 17 GB of weights does not go into 16 GB of RAM, and no context setting changes that. Adding swap does not rescue it either. Ollama memory-maps the GGUF file, so once the resident pages exceed RAM the kernel starts evicting and re-reading them, and every token then pulls gigabytes off the disk. The box sits at high iowait and produces well under one token per second.

32 GB is the entry point. Weights take 17 GB, the vision projector adds about 0.9 GB, and you have roughly 12.5 GB left, which covers about 128k tokens of f16 context with a margin. Q8_0 weights at 29 GB fit within 32 GB, but usable context also depends on the vision projector, cache, operating system and runtime memory. Check the total footprint on your box before treating Q8 as runnable on this tier.

64 GB is comfortable. Q4 leaves room for the full 256k window, and Q8_0 weights fit with the full 256k behind them too. Before paying for 64 GB to get Q8, be clear about what you are buying: slightly better output at half the speed, on a machine that was already slow. Q4 already reaches 128k tokens on a 32 GB plan, and that is the better trade for almost everyone.

How fast is CPU inference on a VPS?

Generating one token from a dense model means reading every weight from memory once. Not some of them. All of them. So the speed limit is not your core count, it is memory bandwidth divided by the size of the weights. At Q4 that is 17 GB of memory traffic per token.

[
  {
    "label": "DDR4-2666, 2 channel",
    "mem_bandwidth_gb_s": 42.6,
    "ceiling_tok_s": 2.5
  },
  {
    "label": "DDR4-3200, 2 channel",
    "mem_bandwidth_gb_s": 51.2,
    "ceiling_tok_s": 3
  },
  {
    "label": "DDR5-4800, 2 channel",
    "mem_bandwidth_gb_s": 76.8,
    "ceiling_tok_s": 4.5
  },
  {
    "label": "DDR4-3200, 8 channel",
    "mem_bandwidth_gb_s": 204.8,
    "ceiling_tok_s": 12
  },
  {
    "label": "DDR5-4800, 12 channel",
    "mem_bandwidth_gb_s": 460.8,
    "ceiling_tok_s": 27.1
  }
]

Those are ceilings, not measurements. Real output lands at roughly 50 to 70 percent of the figure shown, because memory latency and imperfect prefetching mean you never reach the theoretical peak. A two-channel DDR4-3200 VPS has a ceiling of 3 tokens per second, so expect about 2. A two-channel DDR5-4800 box has a ceiling of 4.5, so expect about 3.

The large server rows come with a warning. A twelve-channel EPYC platform has 460.8 GB/s and a ceiling of 27.1 tokens per second, but you do not rent a whole EPYC. Memory bandwidth is a host-wide resource shared by every tenant on that machine, so an 8 vCPU slice does not come with twelve channels of exclusive bandwidth. GPU-focused guides skip this entirely, and it is the reason two VPS plans with identical vCPU counts can differ by a factor of three on the same model.

More vCPUs stop helping early for the same reason. Once the cores request data faster than the memory controller can deliver it, extra threads add scheduling overhead and nothing else. Ollama has no server-wide environment variable for the thread count, so set num_thread in the request options to your physical core count, measure, then try half that number. On many shared plans the lower setting is faster.

Prompt processing behaves differently. Prefill, the pass over your input before the first token appears, is compute bound rather than bandwidth bound, so it does scale with cores. The practical effect is a long pause before output starts on a large prompt, followed by the slow steady rate above. Time both halves separately with --verbose, which prints a prompt eval rate and an eval rate for every request.

If the dense 27B is simply too slow, look at the qwen3.6:35b-a3b tags before giving up on CPU. Those activate roughly 3 billion parameters per token instead of all 27.3 billion, so the memory traffic per token falls by close to an order of magnitude even though the file on disk is larger. You trade RAM footprint for speed. Runtime choice matters here too, and Ollama and llama.cpp expose different CPU tuning controls over the same underlying inference code.

When to rent a GPU hour instead

[
  {
    "label": "L40S, 48 GB",
    "mem_bandwidth_gb_s": 864,
    "ceiling_tok_s": 51
  },
  {
    "label": "RTX 4090, 24 GB",
    "mem_bandwidth_gb_s": 1008,
    "ceiling_tok_s": 59
  },
  {
    "label": "A100, 80 GB",
    "mem_bandwidth_gb_s": 2039,
    "ceiling_tok_s": 120
  },
  {
    "label": "H100 SXM, 80 GB",
    "mem_bandwidth_gb_s": 3350,
    "ceiling_tok_s": 197
  }
]

The same formula applied to published GPU memory bandwidth gives a different category of answer. A 24 GB consumer card has a ceiling of 59 tokens per second on these weights. A current data centre card reaches 197. That is not a gap you close by tuning thread counts. The card runs its memory at 1008 GB/s where your VPS runs at tens.

So draw the line by workload rather than by preference. CPU inference is the right answer when the work is asynchronous and nobody is waiting on it: overnight summarisation of a document pile, or a nightly classification job that runs while you sleep. Rent a GPU the moment a person is waiting for output, or the moment requests arrive faster than your box can finish them. CPU runtimes are not limited to one request at a time: the llama.cpp server documents parallel decoding and continuous batching on CPU as well as GPU, and Ollama can serve parallel requests too, at the cost of more cache. What decides it is your measured serving rate against your arrival rate, so time a realistic batch on your own box. Once requests arrive faster than that, the queue grows.

The cost comparison is less obvious than it looks. A 64 GB VPS bills every hour of the month whether the model is loaded or not, while an hourly GPU rental separates compute charges from storage charges that can continue after it stops. If your real usage is two hours a day, the rented GPU can be both faster and cheaper. Work out your duty cycle first, then price it. Picking a VPS with a GPU covers what to check on the instance itself, and vLLM pulls ahead of Ollama once you serve concurrent requests on a GPU because it batches them properly.

There is a third option people forget. Keep the 27B on CPU for batch work and put a hosted API model in front of the interactive path. Nothing requires one model to serve both.

Install Ollama and measure your own box

The install script is the official one, and it sets up a systemd service running as a dedicated ollama user.

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
free -g

ollama --version should print 0.32.12 or later, the first release with Qwen 3.8 27B support. Check free -g before you pull anything. It reports GiB, so a plan advertised with 32 GB of RAM can show about 29 in the total column; compare the available memory with the weights and cache budget above.

Set the runtime options in a systemd override rather than in your shell. The model runs inside the service, so it never sees your interactive environment.

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=60m"

sudo systemctl restart ollama
ollama pull qwen3.8:27b
ollama run qwen3.8:27b --verbose "Name two Linux distributions."

The --verbose output is the measurement you came for. eval rate is your tokens per second during generation. prompt eval rate is your prefill speed. load duration is how long it took to read the weights off disk, which is why OLLAMA_KEEP_ALIVE=60m is set: on CPU, reloading 17 GB from disk on every request costs more than the request itself. The default idle timeout is five minutes, short enough that a batch queue with gaps between items pays that load cost over and over, and the options for keeping a model resident cover both the per-request keep_alive field and making the setting survive a reboot.

While the model is loaded, check the footprint from a second terminal.

ollama ps

The SIZE column is the real memory footprint including the KV cache, and it should sit close to the weights plus the vision projector plus the row for your context length in the KV chart. At 8192 tokens with an 8-bit cache, the growing part of the cache is only about a quarter of a gigabyte, against 0.5 GB if it stayed at f16, with the fixed linear-attention state on top. The PROCESSOR column should read 100% CPU. If it reads anything else, something has claimed a GPU and the speed numbers in this guide do not describe your box.

Failure modes and the exact strings you will see

The model refuses to load. Ollama prints a line naming both figures, in the shape model requires more system memory (18.6 GiB) than is available (15.2 GiB). This is the good failure, because Ollama checked before allocating rather than letting the kernel sort it out. Lower the context length, drop to a smaller tag, or move to a larger plan.

The process disappears in the middle of an answer. The client shows nothing useful, and journalctl -u ollama -n 50 shows the service restarting. Run dmesg -T | tail, and a line reading Out of memory: Killed process ... (ollama) means the kernel OOM killer took it. That happens when the pre-load check passed but the cache grew past the estimate during a long conversation. Cut the context length.

The pull fails at once. Error: pull model manifest: file does not exist means the tag is not in the library. A version number that has not shipped produces exactly this, and so does any typo in the tag. qwen3.8:27b itself is real, so if that exact tag fails, look for a typo first. Confirm the tag on the library page before blaming your network, and if a real tag pulls but will not run, check ollama --version against 0.32.12.

Everything works but it is unbearably slow. Under one token per second on a box with enough RAM points at paging rather than at compute. Run vmstat 1 while generating. A non-zero si or so column means the kernel is swapping, and the fix is less context or fewer loaded models. Steady high wa with no swap activity means the memory-mapped weights are being re-read from disk, which means they do not really fit.

The first token takes 30 seconds and then output speeds up. That is prefill, and it is normal. A long system prompt is paid for on every request that misses the cache, so shorten the system prompt before tuning anything else.

What a CPU-only 27B is actually good for

Set expectations from the numbers rather than from hope. At two to four tokens per second, a 500 token answer takes between two and four minutes. That is unusable for chat and perfectly workable for a queue. A model that thinks before it replies makes that arithmetic worse, because the hidden reasoning tokens are generated at the same slow rate as the answer, so matching the reasoning effort level to the job is one of the few levers that shortens a reply without swapping the model. Document summarisation, bulk tagging, field extraction from a backlog of files, and unattended code review all tolerate it, because nothing is waiting on the reply. Coding assistance sits right on that line, so pointing a coding agent at a model you host pays off for background jobs like commit messages and test scaffolding, and not for the inline suggestions you sit and wait for.

The privacy argument is the real one. The model runs on hardware you rent and control, no request leaves the box, and there is no per-token bill. That is worth a lot for regulated data even at three tokens per second. Weigh it honestly against the alternative: self-hosting a frontier-scale model needs an order of magnitude more hardware, and a 27B on CPU is the cheapest point on that curve where the output is still worth reading.

To benchmark any of that you need real structured input, and most public data APIs want an account before you can even measure throughput. Strasmore's demo schema (we run it) documents read-only SQL over US equities and options data with no key and no signup, using database-qualified table names: https://ai.strasmore.com/api/demo/sql?sql=SELECT ticker, close FROM global_markets.delayed_stocks_minute_aggs LIMIT 5. The demo limits are 500 rows and 20 seconds per call, with history from 1 January 2025 onward, comfortably more than a two-tokens-per-second box will consume. The full column list is at https://ai.strasmore.com/api/demo/schema.

If this is your first Ollama install, the full walkthrough for running Ollama on a VPS covers the service setup, the HTTP API and the firewall rules this guide assumes you already have. Do not expose port 11434 to the internet. Ollama's local API requires no authentication, so anything that reaches the port can use your model.

FAQ

Is there a Qwen 3.8 27B model on Ollama?

Yes. As of 21 October 2026 the Ollama library has a qwen3.8 entry, and qwen3.8:27b (also tagged latest) is an 18 GB Q4_K_M build of a 27.3 billion parameter dense model, with a vision projector and a 256K context window. It runs locally and needs Ollama v0.32.12 or later. The older qwen3.6:27b and qwen3.5:27b tags are still listed. Check https://ollama.com/library/qwen3.8/tags for the current list. A tag that does not exist fails with Error: pull model manifest: file does not exist.

How much RAM do I need to run a Qwen 27B model on a VPS?

32 GB is the practical minimum for Q4_K_M. The weights are 17 GB, the vision projector adds about 0.9 GB, the operating system needs around 1.5 GB, and the KV cache adds roughly 1 GB for every 16,000 tokens of context at f16, because only 16 of the model's 64 layers keep a growing cache. A 16 GB plan cannot hold the weights at all, and swap does not help because the file is memory-mapped and the kernel just re-reads it from disk on every token. 64 GB gives you room for the full 256K window, even with Q8_0 weights at 29 GB.

How many tokens per second will a 27B model give me on CPU?

Divide your memory bandwidth by the size of the weights, then take 50 to 70 percent of that. A two-channel DDR4-3200 VPS has a ceiling near 3 tokens per second and delivers about 2. A two-channel DDR5-4800 box has a ceiling near 4.5 and delivers about 3. Higher-channel server platforms look much better on paper, but memory bandwidth is shared across every tenant on the host, so measure your own with ollama run qwen3.8:27b --verbose and read the eval rate line.

Should I use Q4 or Q8 on a CPU-only VPS?

Q4_K_M, in almost every case. Q8_0 is 29 GB against 17 GB, so its weights fit within 32 GB, but the projector, cache, operating system and runtime leave little headroom; 64 GB gives it room for long context. It moves nearly twice as much memory per token, which cuts your tokens per second roughly in half. The quality difference between Q4_K_M and Q8_0 on a 27B model is small for most tasks. Spend the RAM on a longer context instead, since that changes what the model can do rather than how it phrases things.

When is renting a GPU cheaper than a big RAM VPS?

When your duty cycle is low or a person is waiting. A GPU with 24 GB of memory has a theoretical ceiling of roughly 59 tokens per second on these weights, against an estimated 2 or 3 on a typical VPS; hourly GPU compute charges stop when the instance stops, but persistent storage can still cost money. A 64 GB VPS bills all month whether the model is loaded or not. Work out how many hours a day you really generate tokens. The break-even point depends on the rental rates, runtime and persistent storage charges. Continuous low-priority batch work is where the always-on VPS wins.