What it takes to self-host Kimi K3

What it takes to self-host Kimi K3

Self-hosting Kimi K3 means finding room for 2.8 trillion parameters. Moonshot published the open weights in MXFP4, which is about half a byte per weight, so the weights alone come to roughly 1.4 TB before you allocate a single token of cache. No accelerator on sale today holds that on its own. For one GPU the answer is no. For one server the answer is yes in two shapes. The first is an 8-GPU node of 288 GB accelerators, which is what the smallest published multi-user serving recipes use. The second is one machine with about 1.6 TB of combined RAM and VRAM running Unsloth's lossless Q8 GGUF in llama.cpp, with or without a GPU, serving one user at a time.

That is the verdict. Everything below is the arithmetic behind it, because the arithmetic is the part you reuse on the next release. Several infrastructure vendors published K3 deployment guides in the weeks after the announcement on 16 July 2026, and each one assumed you already owned a cluster. This page starts from the other end: what it costs, what you can run instead, and how to tell which of those two you are in.

Total parameters and active parameters are not the same number

K3 is a mixture of experts model. MoE (mixture of experts) splits the network into many sub-networks and lets a router pick a few of them per token. The model card lists 2.8T total parameters and 104B activated per token, from 896 routed experts of which 16 fire for any given token, across 93 layers.

Those two parameter counts answer different questions, and swapping them is the most common mistake in every "can I run this" thread.

Active parameters set the compute cost. A token multiplies through about 104B parameters, so the throughput you should expect resembles a 104B dense model rather than a 2.8T one. That is the entire reason to build an MoE.

Total parameters set the memory cost. The router may select any expert on any token, so every expert has to be stored somewhere fast before the first request arrives. Holding only the hot experts in VRAM and fetching the rest on demand does exist: llama.cpp's --moe-cache-mib keeps a GPU cache of experts that otherwise live in system RAM and uploads the missing ones as the router selects them. It does not shrink the total, though. The rest of the experts still have to sit in RAM, and a PCIe link moves tens of gigabytes per second against the terabytes per second of HBM, so every cache miss costs time. Streaming experts from NVMe because the model does not fit in RAM either is slower again, and Unsloth's K3 guide warns that disk offloading runs much slower.

So it is cheap to compute and expensive to store. Size the hardware against 2.8T. Size your speed expectations against 104B.

Bytes per weight, and where the terabytes come from

Parameter count times bytes per weight. For the weights, that is the whole formula.

[
  {
    "label": "bf16",
    "bytes_per_weight": 2,
    "weights_tb": 5.6
  },
  {
    "label": "fp8",
    "bytes_per_weight": 1,
    "weights_tb": 2.8
  },
  {
    "label": "4-bit (MXFP4, as shipped)",
    "bytes_per_weight": 0.5,
    "weights_tb": 1.4
  },
  {
    "label": "2-bit",
    "bytes_per_weight": 0.25,
    "weights_tb": 0.7
  }
]

K3 was trained quantisation aware and released with MXFP4 weights and MXFP8 activations, so the 4-bit row is the real one. The rows above it are there for scale: at bf16 the same model would want 5.6 TB. MXFP4 also stores one shared 8-bit scale for every block of 32 weights, which adds about 6 percent, and Moonshot's config leaves attention, the shared experts and the vision tower out of the 4-bit format, so the full checkpoint comes to about 1.56 TB by Unsloth's count rather than a clean 1.4 TB.

This narrows the usual escape hatch without closing it. The released checkpoint is already 4-bit, so the easy saving has been taken, but going lower still moves the number. As of 27 October 2026, the dynamic GGUF quantisations of K3 that Unsloth has measured run from 861.3 GB for the 2-bit UD-Q2_K_XL down to 594 GB for the 1-bit UD-IQ1_S, and Unsloth publishes what that costs: top-1 token agreement with the full model is 90.390 percent at UD-Q2_K_XL, 84.127 percent at the 711.1 GB UD-IQ2_XXS and 78.875 percent at UD-IQ1_S. Its Hugging Face repo also carries smaller 1-bit files, down to 466 GB for UD-Q1_0, with no agreement figure published alongside them. That is a measured accuracy loss, and even the smallest file is still about 1.6 times the largest single card.

How many GPUs does Kimi K3 need

[
  {
    "config": "H100 80GB",
    "hbm_per_gpu_gb": 80,
    "gpus_for_weights": 18
  },
  {
    "config": "H200 141GB",
    "hbm_per_gpu_gb": 141,
    "gpus_for_weights": 10
  },
  {
    "config": "B200 192GB",
    "hbm_per_gpu_gb": 192,
    "gpus_for_weights": 8
  },
  {
    "config": "GB300 288GB",
    "hbm_per_gpu_gb": 288,
    "gpus_for_weights": 5
  }
]

Read those as a floor, not a target. They count weights only: no KV cache, no activation buffers, no allocator fragmentation, and no room for a second concurrent request. The split itself does not need to divide evenly: vLLM's pipeline parallelism splits the model along layers and supports uneven splits, and SGLang's K3 prefill recipes run 8 and 16 pipeline stages over 93 layers.

Published guidance sits above the floor. Moonshot's launch guidance recommended a supernode of 64 or more accelerators. As of 27 October 2026 the SGLang cookbook ships an H100 configuration built from four 8-GPU nodes, 32 GPUs and 2,560 GB of aggregate memory, against a floor of 18 cards, and it also lists single-node recipes on one 8-GPU B300 node and one 8-GPU MI350X or MI355X node, 2,304 GB at 288 GB per card. That gap over the floor is not waste. It is KV cache, activation memory, and the headroom that lets the server batch many requests at once. The friendliest row, 5 GB300 class cards, rounds up in practice to exactly that kind of 8-GPU node, which is the smallest machine anyone publishes a multi-user K3 serving recipe for.

The KV cache is the part that surprises people

Weights are a fixed cost. The KV (key value) cache is not: it grows with context length and again with every concurrent user. For ordinary attention the formula is bytes per token = 2 * layers * kv_heads * head_dim * bytes_per_element, and then you multiply by context length and by concurrency.

Here is a worked example, and it is only an example: 64 layers, 8 KV heads, head dimension 128, fp8. That gives 2 64 8 128 1 = 131,072 bytes, so 128 KiB per token.

[
  {
    "label": "8k context",
    "kv_gib_per_user": 1
  },
  {
    "label": "32k context",
    "kv_gib_per_user": 4
  },
  {
    "label": "128k context",
    "kv_gib_per_user": 16
  },
  {
    "label": "1M context",
    "kv_gib_per_user": 128
  }
]

One user at 128k context costs 16 GiB. One user at the full million costs 128 GiB, which is nearly half of a 288 GB card for one conversation, before a single weight is loaded.

K3 does not use ordinary attention, and that last number is why. Its 93 layers are 69 KDA (Kimi Delta Attention) layers and 24 Gated MLA (multi-head latent attention) layers. KDA keeps a fixed size recurrent state instead of a cache that grows with every token, and MLA compresses key and value into one low rank latent vector, so the real per-token cost falls a long way below the worked example. Moonshot's published config.json gives the size of that latent: a kv_lora_rank of 512 plus a 64-wide rotary key (qk_rope_head_dim), so 576 values per token per MLA layer. Across 24 layers at 2 bytes each that is 27,648 bytes, about 27 KiB per token, or roughly 27 GiB for a full 1,048,576 token conversation, plus the fixed KDA state. Treat that as an estimate rather than a measurement: engines pad and lay out the cache in their own ways, and an 8-bit cache roughly halves it.

To see what your own deployment holds, do not watch nvidia-smi while raising --max-model-len. vLLM preallocates its KV cache pool, sized from --gpu-memory-utilization (0.92 by default in the current docs) or from an explicit per-GPU cache budget, so total GPU memory shows the pool rather than one conversation. Read the startup log instead, where vLLM reports the KV cache size in tokens and the maximum concurrency at your context length.

The shape of the reasoning survives the next release. If a model advertises a million token context and says nothing about its attention design, assume the cache is the binding constraint until someone proves otherwise.

Tier 1: rent the cluster by the hour

This is the tier that serves K3 at its shipped precision to many users at once. One machine with enough RAM can also hold K3 at its released precision through Unsloth's Q8 GGUF, which Unsloth labels full precision lossless, but that route serves one user at a time and is covered in tier 2. Here you do not buy the hardware. You rent it for the hours you need it and you stop it afterwards.

[
  {
    "label": "1 GPU, always on",
    "gpu_hours": 720,
    "usd_cost": "1,800"
  },
  {
    "label": "8 GPUs, 4 hours a day",
    "gpu_hours": 960,
    "usd_cost": "2,400"
  },
  {
    "label": "8 GPUs, always on",
    "gpu_hours": 5760,
    "usd_cost": "14,400"
  },
  {
    "label": "32 GPUs, always on",
    "gpu_hours": 23040,
    "usd_cost": "57,600"
  }
]

The rate is an assumption, not a quote. On-demand rates vary widely by provider and by GPU generation, the 288 GB parts that the single-node recipes need cost more per hour than an H100, and reserved capacity is cheaper. Take your provider's real number and redo the multiplication: GPUs times hours times rate. The point of the chart is the ratio. Bursting an 8 GPU node for four hours a day costs 2,400 USD a month, while leaving the SGLang sized 32 GPU H100 configuration running costs 57,600 USD.

Both mainstream servers publish a launch command on the model card.

pip install vllm
vllm serve "moonshotai/Kimi-K3"

pip install sglang
python3 -m sglang.launch_server --model-path "moonshotai/Kimi-K3" --host 0.0.0.0 --port 30000

Neither bare command is what you run on a real cluster. Add the parallelism flags that match your hardware: SGLang takes --tp-size for tensor parallel, --pp-size for pipeline parallel and --ep-size for expert parallel. Tensor parallel times pipeline parallel has to equal the GPUs you actually have, and expert parallel splits the experts within that group, which is why the 32 GPU H100 recipe sets both --tp-size 32 and --ep-size 32.

Check the server came up before you send real traffic:

curl http://127.0.0.1:30000/v1/models

A healthy server answers with a JSON object listing the model id. Connection refused means the process is still loading weights or has already exited, so read the server log before you retry.

The common failure in the first weeks was a runtime older than the model. K3 shipped with KDA and a new MoE layer that the stable vLLM and SGLang releases did not carry at launch, and the symptom is the server exiting during startup with a line of the form Model architectures [...] are not supported for now. No config change fixes that, because the code to run those layers is not in your build. That has since changed: as of 27 October 2026 SGLang has carried K3 support in its releases since v0.5.17, with v0.5.22 the latest, and vLLM's v0.28.0 to v0.32.0 release notes all include K3 work. Install a current release. Some hardware-specific paths still name a dedicated image, for example SGLang's NVFP4 builds on Blackwell, so check the recipe for your card.

One cost note that catches people. The meter starts when the instance starts, not when the model is ready. A 1.56 TB download at 1 GB/s is about 26 minutes of cluster time before the first token. Stage the weights on a volume that outlives the instance, so the second run starts in minutes.

Tier 2: run a smaller model on one accelerator

You are not running K3 in this tier. Say that out loud before you start, because most "run K3 locally" threads end here without admitting it.

The fit rule is the same formula in miniature: parameters times bytes per weight, plus KV cache, plus about 2 GB of runtime overhead, has to sit under your VRAM. At 4-bit that is roughly half a byte per parameter, which gives comfortable pairings:

• 16 GB card: a 7B model at 4-bit with room for a long context
• 24 GB card: a 14B model at 4-bit
• 48 GB card: a 32B model at 4-bit
• 80 GB card: a 70B model at 4-bit, or a 30B class MoE at 8-bit

Every pairing above assumes one request at a time, and the moment a second person sends a prompt each concurrent slot wants its own KV cache, which is the trade Ollama's NUM_PARALLEL and MAX_QUEUE settings make for you between parallel slots, queued requests and the VRAM you have left.

There is one way to run K3 itself on a single large machine, and it sits between this tier and the last. Unsloth documents GGUF builds of K3 in llama.cpp, from 1-bit and 2-bit quantisations up to UD-Q8_K_XL, which it labels full precision lossless, and names a NVIDIA DGX Station as hardware that can run them. Its rule of thumb is RAM plus VRAM roughly equal to the quant size, from about 610 GB for the 1-bit UD-IQ1_S build to about 1.6 TB for the Q8, and below that the model falls back to disk and slows sharply. A GPU is optional: Unsloth's build instructions switch CUDA off "if you don't have a GPU or just want CPU inference". The 1-bit and 2-bit builds are K3 heavily quantised, at the accuracy cost measured above. The Q8 build is K3 at its released precision, served to one user at RAM speed. Neither is a smaller model.

Ollama is the shortest path to a working server on a VPS with a GPU attached:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:14b

ollama run downloads the model on first use, then drops you at a prompt. A tag that does not exist returns Error: model "..." not found, so copy tags from the library page rather than typing them from memory. The full walkthrough, including the systemd unit and remote access, is in running Ollama on a VPS.

llama.cpp gives you more control over quantisation and offload:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

./build/bin/llama-server -m model.gguf -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

-ngl 99 asks for every layer on the GPU. Read the load log: it prints how many layers were offloaded. Layers that spill into system RAM run at RAM bandwidth instead of HBM bandwidth, so generation speed drops by an order of magnitude the moment the model stops fitting. The trade-offs between the two tools are covered in Ollama and llama.cpp side by side.

Tier 3: hosted API, self-hosted orchestration

[
  {
    "label": "Input, cache hit",
    "usd_per_million_tokens": "0.30"
  },
  {
    "label": "Input, cache miss",
    "usd_per_million_tokens": "3.00"
  },
  {
    "label": "Output",
    "usd_per_million_tokens": "15.00"
  }
]

The endpoint is OpenAI compatible, so an existing client works after you change the base URL.

curl https://api.moonshot.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MOONSHOT_API_KEY" \
  -d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "Say hello."}]}'

A working key returns a JSON object with a choices array. A 401 means authentication failed: check that the key is correct and has not expired or been disabled, and that the header reads Authorization: Bearer followed by the key. A model-not-found error usually means the id changed, because providers retire ids between checkpoints.

Now the break-even, using the assumed rental rate above. An always-on 8 GPU node costs 14,400 USD a month, and at 15.00 USD per million output tokens the same money buys about 960 million output tokens from the API. To win on cost you have to generate close to a billion output tokens a month, roughly 30 million a day, and keep the cluster busy the whole time, because idle GPUs bill at the same rate as busy ones. Prompt-heavy agent workloads move the line further away still: repeated context bills at the cache-hit rate of 0.30 USD per million rather than the cache-miss rate of 3.00 USD.

What you self-host in this tier is everything around the model: a gateway that holds the API key so it never reaches a client, request and response logs, retries, rate limits, and per-user budgets. That runs on a small VPS with no GPU at all. The same split applies to closed weights, where self-hosting Claude is not possible at the model level and orchestration is the only part you own.

Which serving stack belongs to which tier

vLLM and SGLang class servers belong to tier 1. They exist to serve many requests at once, with continuous batching and a paged KV cache, plus tensor and expert parallelism spread across several nodes. They assume datacentre accelerators and a fast interconnect between them. On a single consumer card they are heavier to install and give you little you would notice.

llama.cpp and Ollama belong to tier 2. They target one machine, GGUF quantisation, CPU offload when the model does not fit, and low concurrency. llama.cpp will load an enormous MoE by keeping expert weights in system RAM (--cpu-moe or --n-cpu-moe) and, with --moe-cache-mib, caching the experts it uses on the GPU, and that is how Unsloth's K3 GGUF builds run on a single machine. Speed then rides on RAM bandwidth and on how often the router misses the cache, and if the file is larger than RAM plus VRAM it drops to disk. It is a workable setup for one user. It is not a service you can put many users on. The full comparison is in Ollama against vLLM, and it does not change with the model: the question is always whether you are serving many users on shared hardware or one user on your own.

The four numbers that outlive this checkpoint

1. Total parameters times bytes per weight gives the memory floor. Nothing runs below it at the shipped precision, and quantising below a 4-bit release buys you up to about 70 percent at an accuracy cost, not an order of magnitude.
2. Active parameters give the throughput class. A 2.8T MoE with 104B active computes like a 104B model.
3. KV cache per token, times context length, times concurrency, is the cost that keeps growing after you have paid for the weights.
4. Tokens per second per dollar is the only number that picks a tier. Everything above is input to it.

Apply those four to any release and you get the right answer before you open a vendor guide. The next one is already announced: Moonshot rolled out K3.1 Pro, built on K3 with further post-training, in Kimi Code on 23 October 2026, and its API quickstart says the full weights will be released by 4 November. Then date every figure you write down. Prices and supported-architecture lists both moved inside the two weeks after K3 launched. The launch figures on this page were published in July 2026, and the serving recipes, runtime releases and quantisation numbers were rechecked on 27 October 2026.

FAQ

Can I run Kimi K3 on a single GPU?

No. The weights are about 1.4 TB at the MXFP4 precision Moonshot ships, and even the smallest published quantisation of K3, Unsloth's 466 GB UD-Q1_0 file, is about 1.6 times the 288 GB on the largest widely available accelerators. llama.cpp can keep experts in system RAM and cache the active ones on the GPU, but that still needs roughly the quant size in RAM plus VRAM, and below that the model drops to disk and runs much slower. The smallest published serving recipes use one 8-GPU node of B300 or MI350X/MI355X cards, and the H100 recipe uses 32.

How much VRAM does Kimi K3 need?

Start at 1.4 TB for weights alone, which is 18 H100 80GB cards or 5 GB300 class cards. Then add KV cache and activation memory on top. Moonshot's launch guidance recommended 64 or more accelerators. As of 27 October 2026 the SGLang cookbook publishes single-node recipes on 8 B300 or 8 MI350X/MI355X cards, 2,304 GB aggregate, and a 32 GPU H100 configuration with 2,560 GB aggregate, so treat the weights figure as a floor rather than a requirement.

Does quantisation make Kimi K3 fit on one node?

The shipped 4-bit checkpoint already fits on one node if that node is eight 288 GB cards, which is what SGLang's B300 and MI355X recipes use, and on one machine with about 1.6 TB of RAM plus VRAM through Unsloth's lossless Q8 GGUF. Going lower helps smaller machines: Unsloth's measured dynamic GGUF builds range from 861.3 GB at 2-bit to 594 GB at 1-bit, and its repo lists 1-bit files down to 466 GB, against about 1.56 TB lossless. The accuracy cost has been measured: top-1 agreement with the full model falls to 90.390 percent at UD-Q2_K_XL and 78.875 percent at UD-IQ1_S. None of them fits on a single card.

Is renting GPUs cheaper than the Kimi K3 API?

Only at high and steady volume. At an assumed 2.50 USD per GPU hour, an always-on 8 GPU node costs 14,400 USD a month, and the same money buys about 960 million output tokens at the published rate of 15.00 USD per million. You also pay for idle hours, weight downloads and the person who keeps the cluster alive. Rent by the hour for bursts, and compare against your own measured token volume instead of a guess.

What does 104B active parameters mean for speed?

It means the arithmetic per token is that of a 104B model, so throughput lands in that class rather than the 2.8T class. It says nothing about memory: all 2.8T parameters stay resident, because the router can call any expert on any token. Use the active count to predict tokens per second, and the total count to size VRAM.