Self-host Qwen3.8-Flash-Next: what it takes

What does it take to self-host Qwen3.8-Flash-Next?

There are three ways to self-host Qwen3.8-Flash-Next. A server with enough RAM for a 2-bit or 3-bit GGUF file can run it on llama.cpp, even with no GPU. A PC with a 12 GB graphics card can run it through Strata, a separate engine built for this one model. Serving it to many users at near its published precision takes four or more datacenter GPUs on vLLM or SGLang.

This guide explains where each of those floors comes from. The figures below come from the official model card, the quantizer's card, the Strata README, and the vLLM and SGLang recipes. All of them were read on 9 October 2026. The model is new and its tooling changes every day, so compare those dates with what you download.

What is Qwen3.8-Flash-Next?

Qwen3.8-Flash-Next is a MoE (mixture of experts) language model from the Qwen team. Each MoE layer holds many expert blocks, and a small router picks a few of them for each token. Only the chosen experts run, so the work per token is far smaller than the total parameter count.

The official card describes the size as "125B with 6B activated, plus 51B n-gram embedding and 4B MTP". Its type line reads "Causal Language Model with Vision Encoder". It accepts images, and the card's API examples also send a video as an MP4 file. The output is always text.

The native context window is 262,144 tokens, and it can be extended to 1,000,000. Going past the native length uses YaRN, a method that rescales the model's position encoding. The card warns that static YaRN scaling can hurt performance on shorter texts. It advises changing the RoPE (rotary position embedding) settings only when you really need long context.

The card calls the release an experimental preview of the architecture that will underpin Qwen4. That makes it a preview of an architecture. It is not Qwen4 itself.

Why do people quote 125B, 177B and 180B?

[
  {
    "label": "Main MoE backbone",
    "billions": 125
  },
  {
    "label": "N-gram embedding",
    "billions": 51
  },
  {
    "label": "MTP layers",
    "billions": 4
  }
]

The totals differ because they count different parts. 125B is the main MoE backbone only. The n-gram embedding adds 51B, and the MTP (multi-token prediction) layers add 4B. Add the three together and you get 180B. That is the total the Hugging Face Hub shows for the stored weights.

The 177B figure has no primary source behind it. None of the Qwen pages behind this guide state it, so treat it as unofficial. Tallies that leave out a component produce smaller totals, which is why several numbers circulate for one model.

Two numbers matter in practice. The 6B activated parameters decide how much work each token costs. The size of the file you download decides how much memory you need.

What does the Qwen community licence allow?

The card lists the licence as qwen-community-1.0. It allows broad use but attaches conditions, so read the full text on the model's Hugging Face page before you build anything on it. These are the conditions that matter for a self-hosted server:

• A commercial product or service with more than 100,000,000 monthly active users or US$20,000,000 monthly revenue must prominently display the model name in its user interface.
• If you or an affiliate run a Model as a Service or AI Work Assistant business, you need a separate licence from Qwen before you use the model.
• Model as a Service means giving a third party access to language model inference or fine-tuning, for example through an API or a hosted endpoint. An AI Work Assistant is an independent product built mainly for AI-assisted coding or office productivity.
• The separate-licence rule does not apply to internal use, as long as the software, its outputs and its model capabilities are not made available to third parties.
• The software and its output come "AS IS", with no warranty.

In practice, a private server that only you or your own team use sits inside the internal-use exemption. Selling API access to that server to other people is the case that needs a separate licence from Qwen. Because of these conditions, describe the model as released under a community licence with use conditions, not as unrestricted open source. This summary is not legal advice.

Route one: GGUF files on llama.cpp

GGUF is the single-file model format that llama.cpp reads. Quantization stores each weight in fewer bits, so the file shrinks at some cost in quality. ISTA-DASLab publishes GSQ-RCO quantizations of this model, and its card lists 4 sizes. Each one is split into two shards. Shard 1 holds the transformer weights. Shard 2 holds the n-gram table.

[
  {
    "label": "Q2_0",
    "bits_per_weight": 2.4,
    "weights_shard_gb": 37.6,
    "ngram_shard_gb": 28.8,
    "total_gb": 66.4
  },
  {
    "label": "IQ2_XS",
    "bits_per_weight": 2.5,
    "weights_shard_gb": 39.2,
    "ngram_shard_gb": 28.8,
    "total_gb": 68.0
  },
  {
    "label": "IQ3_XXS",
    "bits_per_weight": 3.0,
    "weights_shard_gb": 47.0,
    "ngram_shard_gb": 28.8,
    "total_gb": 75.8
  },
  {
    "label": "IQ3_S",
    "bits_per_weight": 3.5,
    "weights_shard_gb": 54.8,
    "ngram_shard_gb": 28.8,
    "total_gb": 83.6
  }
]

The smallest, Q2_0, totals 66.4 GB. The largest, IQ3_S, totals 83.6 GB. Every size shares the same 28.8 GB n-gram shard, so only shard 1 changes, from 37.6 GB to 54.8 GB. A 0.91 GB vision projector file (the mmproj) serves all four. These sizes are from the card on 9 October 2026. Quantized repositories change, so check the file list on the day you download.

How much RAM do the GGUF files need?

The safe rule is simple. Give the machine more RAM than the total file size, plus the projector, plus the KV cache. The KV (key-value) cache is the attention state the model keeps for each token in the context, so it grows with the context length you set with -c. With that much memory, nothing is read from disk after the model loads. For the general method behind that sum, see how big an LLM fits in your RAM.

The ISTA card describes a lower floor. It says that only the first shard holds transformer weights and needs to be resident. It tells you to run llama.cpp with -lm mmap --lazy-mode on. In llama.cpp, -lm is short for --load-mode, and mmap maps the file into memory instead of copying it. Lazy mode reads rows of certain tensors, such as embedding tables, from disk on demand instead of keeping them resident. It requires mmap. With both flags, the n-gram shard stays on disk. The resident memory is then shard 1, the projector and the KV cache.

That lower floor has a cost. Every row read from the n-gram shard is a disk read, so keep the files on an SSD. The card itself says that keeping the n-gram table in RAM removes the paging cost and is worth it where memory allows.

Why can it run without a GPU at all?

Each generated token runs through the router's chosen experts, about 6B parameters, not the full 125B backbone. A dense model of 125B would make the CPU work through all of its weights for every token. That difference is the reason a CPU-only run is plausible.

Plausible does not mean fast. Nobody behind this guide measured tokens per second for this model on a CPU, so this guide prints no speed figure. Measure it on your own machine. Every response from llama-server's /completion endpoint carries a timings object, and its predicted_per_second field is your generation speed.

What the quant card does not compare

The ISTA card states: "All models are evaluated against the BF16 base model." BF16 (bfloat16) is a 16-bit number format, and it is the tensor type the Hub lists for the official weights. The card makes no comparison with Q4_K_M, the 4-bit llama.cpp quant that many people use as their reference point. The official Qwen card does not state the precision of its own benchmark runs either.

So you can see how far each ISTA file sits from BF16 on ISTA's tests. You cannot read off how a 2-bit or 3-bit file compares with a Q4_K_M build you may have used before. If that trade-off is new to you, what Q4, Q8 and FP16 quantization cost in quality explains it on smaller models. Test the quant you pick on your own prompts before you rely on it.

Pin a llama.cpp build that knows qwen4exp

llama.cpp added this architecture in pull request #27742, "model: add Qwen3.8-Flash-Next (qwen4exp)". The first release whose notes list it is b10660, published on 27 August. The architecture name inside the GGUF files is qwen4exp. A build from before b10660 does not know that architecture, so it cannot load the files. If loading fails, read the error in the server output before you try anything else.

Treat b10660 as a floor, not a target. Many follow-up commits for this model landed after it. The pull request itself notes that its quantized results are not bit-identical to the reference. Also confirm that the build you pin has the load-mode flags the ISTA card uses. Building from a tag looks like this:

sudo apt update && sudo apt install -y build-essential cmake git libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
TAG=b10660   # the floor; replace it with the newest release you have tested
git checkout "$TAG"
cmake -B build
cmake --build build --config Release -j"$(nproc)"
./build/bin/llama-server --help | grep -E 'load-mode|lazy-mode'

The last command should print the help lines for --load-mode and --lazy-mode. Empty output means your tag is older than those flags, so pick a newer one. Pinning llama.cpp releases on a server covers how to record the tag and upgrade on purpose instead of by accident.

Download a quant and start the server

The Hugging Face command-line tool can fetch one directory at a time. Install it in a virtual environment, because Ubuntu 24.04 blocks system-wide pip installs. This example fetches IQ3_XXS, the size the ISTA card uses in its own example, plus the vision projector:

sudo apt install -y python3-venv
python3 -m venv ~/hf-cli
~/hf-cli/bin/pip install -U huggingface_hub
~/hf-cli/bin/hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_XXS/*" "mmproj-*" --local-dir ~/models/qwen38-flash-next
ls -lh ~/models/qwen38-flash-next/IQ3_XXS/

ls should list two shard files, one ending in -00001-of-00002.gguf and one ending in -00002-of-00002.gguf. For another size, change the directory name in --include. Then point llama-server at the first shard, as the card's own example does:

./build/bin/llama-server \
  -m ~/models/qwen38-flash-next/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --mmproj ~/models/qwen38-flash-next/mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  -lm mmap --lazy-mode on \
  -c 16384 --host 127.0.0.1 --port 8080

In another terminal, check that the model has loaded:

curl -s http://127.0.0.1:8080/health

A loaded server answers {"status":"ok"}. While the weights are still loading, it answers with a loading error instead, so wait and try again. The -c 16384 context is a starting value. Raise it only after you see how much memory the KV cache takes at that size.

On a machine with a GPU build, the card's example adds -ngl 99 to put all layers on the GPU. That is where its advice to plan for shard 1 in VRAM (the GPU's own memory) comes from. For the general setup around this server, see running llama.cpp server on a VPS.

What about Ollama?

The ISTA card's only Ollama instruction is ollama run hf.co/<this-repo>, with a note to pick the directory that matches your memory budget. Ollama's releases page, up to v0.40.2 on 8 October 2026, does not mention this model or the qwen4exp architecture. That does not prove Ollama cannot load it. It means no Ollama release note confirms it, so test your Ollama version before you plan around it.

The lower memory floor above also depends on --lazy-mode, which is a llama.cpp flag. Ollama runs models through its own engine and does not take llama.cpp command-line flags. On Ollama, plan for the safe rule: RAM above the full file size. Ollama versus llama.cpp covers that trade in general, and importing a GGUF model into Ollama covers the manual import path.

Route two: Strata, which is not part of Ollama

Strata (github.com/Niko1221/Strata) is an MIT-licensed inference engine built to run Qwen3.8-Flash-Next on consumer PCs. Many people find it by searching for "ollama strata", but it is a separate project. Its README does not mention Ollama. The README also notes that a few parts, and every model, carry their own licences. The Qwen licence above still applies to the weights.

Strata serves a browser chat app at http://127.0.0.1:8080. It exposes an OpenAI-compatible API under /v1 and an Anthropic-style endpoint at /v1/messages. It also serves the OpenAI Responses API at /v1/responses for tools such as Codex CLI.

What hardware does Strata need?

The README states these floors:

• GPU: 12 GB of VRAM or more. Supported cards are the NVIDIA RTX 20 to 50 series and several AMD RX 6000, 7000 and 9000 models.
• RAM: 32 GB or more. The README says 64 GB runs every model size.
• Disk: about 80 GB free. The installer downloads about 70 GB of model data.
• OS: Windows 10 or 11, or Linux, with a current NVIDIA or AMD graphics driver.

Other sources describe lower floors. The v0.1.38 release headline says a 6 GB card can start, and the ISTA card says Strata is reported to run on an 8 GB GPU. Treat 12 GB as the floor the README commits to.

The README also prints tokens-per-second figures for two consumer GPUs and a projection for a third card. Those are the project's own claims. The NVIDIA figures were measured with engine versions 0.1.36 and 0.1.26, which are older than the current tag. Nobody behind this guide measured them, so they are not repeated here. Read them in the README, then measure on your own card.

Pin a Strata tag

Early write-ups about Strata cite v0.1.39, tagged on 4 October 2026. Between 6 and 8 October six more tags followed, and the newest on 9 October was v0.1.41. At that pace, "the latest Strata" means something different every day. The README installs from the main branch, by zip download or git clone. Clone it and check out a tag instead, so you know exactly what you run:

git clone https://github.com/Niko1221/Strata.git
cd Strata
git checkout v0.1.41   # newest on 9 Oct 2026; use the tag you have tested
./setup.sh

setup.sh downloads the model and starts Strata. If the download stops, run ./setup.sh again and it resumes. The README's ./update.sh updates Strata without starting it. Run it only when you mean to move off your pinned tag, and then write down the tag you are on.

Strata listens on 127.0.0.1 by default, which is the right default on a remote GPU server. Reach it through an SSH tunnel from your own computer:

ssh -N -L 8080:127.0.0.1:8080 user@your-server

Then open http://127.0.0.1:8080 in your local browser. The README shows a Windows command that adds --host 0.0.0.0 and an --api-key flag for access from other devices, and it says to always set a key. It gives no Linux equivalent, so the tunnel is the safer choice. Strata needs a real GPU, so on rented hardware you need a VPS with a GPU, not a CPU-only server.

A different project with the same name

ro99/strata on GitHub is a different project with the same name. It is an Apache-2.0 engine written in C++20 and CUDA, and it splits a model checkpoint between GPU memory, host RAM and NVMe storage. Its README lists models such as Gemma 4 31B-IT, DeepSeek-V4-Flash and GLM-5.2. It does not list Qwen3.8-Flash-Next, and it does not mention Niko1221/Strata. If a guide tells you to build "Strata" from C++ sources, check which repository it means.

Route three: vLLM or SGLang on a multi-GPU node

The official card strongly recommends dedicated serving engines such as SGLang and vLLM for production or high-throughput workloads. Both projects publish recipes for this model, and both recipes target datacenter GPUs.

The vLLM recipe requires vLLM 0.29.0 or newer and the dedicated image vllm/vllm-openai:qwen38-flash-next. It says that installing from PyPI is not supported for this recipe, so use the image, not a plain pip install vllm. The FP8 (8-bit floating point) checkpoint is 172.78 GiB, and the BF16 checkpoint is 335.28 GiB. The host also needs at least 51 GB of RAM plus headroom, for the n-gram embedding offload.

The recipe's hardware options show the scale. Tensor parallelism (TP) splits each layer across several GPUs:

• 4x GB300: TP4 is the recommended full-tray setup. TP2 is the minimum FP8 deployment.
• 8x H200: plain TP8 does not work with the FP8 checkpoint, so the recipe uses TEP8 with the Triton MoE backend.
• 4x H100: TP4 with the Triton MoE backend, with PLE CPU offload turned on through VLLM_PLE_CPU_OFFLOAD=1.
• 4x MI355X: TP4 on ROCm with AITER enabled.

This is the recipe's FP8 command for 4x GB300:

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --kv-cache-dtype fp8 \
  --attention-config.indexer_kv_dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

The --speculative-config line is where the MTP layers come in. It turns on speculative decoding with the mtp method: the model proposes up to three tokens ahead, and the main model checks them.

SGLang's cookbook covers single-node TP4 serving on H200, B200, B300 and GB300, and TP8 on AMD MI350X and MI355X. Its rows use FP8 or NVFP4 checkpoints. The H200 row runs TP4 with EP4 (expert parallelism, which spreads the experts across GPUs) on Qwen/Qwen3.8-Flash-Next-FP8, using the image lmsysorg/sglang:qwen38flashnext. The cookbook states no minimum SGLang version, so use the image it names.

This route is for serving a team or a product. It runs FP8 or BF16 checkpoints instead of 2-bit and 3-bit files, and it needs several datacenter GPUs to do it.

Which route fits which machine?

• A CPU-only server with plenty of RAM: route one, llama.cpp with a GGUF file. Plan for RAM above the full file size plus the KV cache. With 128 GB, even IQ3_S at 83.6 GB leaves tens of gigabytes for the KV cache and the OS.
• A server with less RAM but a fast SSD: route one with -lm mmap --lazy-mode on, which keeps the 28.8 GB n-gram shard on disk. Every row the model reads from that shard costs a disk read.
• A machine with one large GPU: route one with a GPU build of llama.cpp and -ngl 99. The ISTA card plans for shard 1 in VRAM, which is 37.6 GB for Q2_0.
• A desktop or GPU server with a consumer card of 12 GB or more and at least 32 GB of RAM: route two, Strata, pinned to a tag.
• A node with four or more datacenter GPUs: route three, vLLM or SGLang with the FP8 checkpoint, for many users at once.
• An existing Ollama setup: test ollama run hf.co/... on your version first. No Ollama release note confirms support yet.

Before you plan hardware around any of these, check whether a smaller model does the job. Running Qwen3.8 27B on a VPS needs far less memory than any file in the chart above.

FAQ

Can Qwen3.8-Flash-Next run on a CPU-only server?

Yes, through llama.cpp and a GGUF file. Only about 6B of its parameters are active for each token, which is why a CPU-only run is plausible. Plan for RAM above the file size plus the KV cache. The smallest ISTA-DASLab file, Q2_0, is 66.4 GB. Use llama.cpp release b10660 or later, because older builds do not know the qwen4exp architecture. Measure the speed yourself with the timings field in llama-server's responses.

Is Strata part of Ollama?

No. Strata (github.com/Niko1221/Strata) is a separate, MIT-licensed engine built for this one model, and its README does not mention Ollama. It serves its own OpenAI-compatible and Anthropic-style APIs on 127.0.0.1:8080. Its README asks for 12 GB of VRAM or more and 32 GB of RAM or more. Do not confuse it with ro99/strata, a different engine with the same name.

How much RAM does the smallest Qwen3.8-Flash-Next GGUF need?

The Q2_0 file from ISTA-DASLab is 66.4 GB in total, as of 9 October 2026. The safe plan is RAM above that, plus the 0.91 GB vision projector, plus the KV cache for your context length. The ISTA card also describes a lower floor. With -lm mmap --lazy-mode on, the 28.8 GB n-gram shard stays on an SSD, and only the 37.6 GB weights shard needs to be resident.

Can I use Qwen3.8-Flash-Next commercially?

The qwen-community-1.0 licence allows broad use with conditions. Products above 100,000,000 monthly active users or US$20,000,000 monthly revenue must display the model name prominently. Running a Model as a Service or AI Work Assistant business needs a separate licence from Qwen. Internal use is exempt when the model, its outputs and its capabilities are not made available to third parties. Read the full licence on the model's Hugging Face page before you ship anything.