By What it takes to self-host Kimi K3
Kimi K3 is 2.8 trillion parameters. Here is the VRAM arithmetic, the KV cache math, and the three honest ways to run it without a 32 GPU cluster.
What it takes to self-host Kimi K3
Self-hosting Kimi K3 means finding room for 2.8 trillion parameters. Moonshot published the open weights in MXFP4, which is about half a byte per weight, so the weights alone come to roughly 1.4 TB before you allocate a single token of cache. No accelerator on sale today holds that on its own. K3 is a multi-node model, and for one server the answer is no.
That is the verdict. Everything below is the arithmetic behind it, because the arithmetic is the part you reuse on the next release. Several infrastructure vendors published K3 deployment guides in the weeks after the announcement on 17 July 2026, and each one assumed you already owned a cluster. This page starts from the other end: what it costs, what you can run instead, and how to tell which of those two you are in.
Total parameters and active parameters are not the same number
K3 is a mixture of experts model. MoE (mixture of experts) splits the network into many sub-networks and lets a router pick a few of them per token. The model card lists 2.8T total parameters and 104B activated per token, from 896 routed experts of which 16 fire for any given token, across 93 layers.
Those two parameter counts answer different questions, and swapping them is the most common mistake in every "can I run this" thread.
Active parameters set the compute cost. A token multiplies through about 104B parameters, so the throughput you should expect resembles a 104B dense model rather than a 2.8T one. That is the entire reason to build an MoE.
Total parameters set the memory cost. The router may select any expert on any token, so every expert has to be resident before the first request arrives. You cannot hold 104B in VRAM and fetch the rest on demand, because the fetch would have to finish in microseconds and a PCIe link moves tens of gigabytes per second. People do try it. Streaming experts from NVMe turns a model that should emit dozens of tokens per second into one that emits a token every few seconds.
So it is cheap to compute and expensive to store. Size the hardware against 2.8T. Size your speed expectations against 104B.
Bytes per weight, and where the terabytes come from
Parameter count times bytes per weight. For the weights, that is the whole formula.
The data behind this chart
[
{
"label": "bf16",
"bytes_per_weight": 2,
"weights_tb": 5.6
},
{
"label": "fp8",
"bytes_per_weight": 1,
"weights_tb": 2.8
},
{
"label": "4-bit (MXFP4, as shipped)",
"bytes_per_weight": 0.5,
"weights_tb": 1.4
},
{
"label": "2-bit",
"bytes_per_weight": 0.25,
"weights_tb": 0.7
}
]K3 was trained quantisation aware and released with MXFP4 weights and MXFP8 activations, so the 4-bit row is the real one. The rows above it are there for scale: at bf16 the same model would want 5.6 TB. MXFP4 also stores one shared 8-bit scale for every block of 32 weights, which adds about 6 percent, so the published repository sits nearer 1.5 TB than a clean 1.4 TB.
This closes the usual escape hatch. "Just quantise it" does not help here, because the released checkpoint is already 4-bit. Going down to 2-bit would bring the weights to 0.7 TB and cost accuracy nobody has measured on this checkpoint. You would still be far past any single card.
How many GPUs does Kimi K3 need
The data behind this chart
[
{
"config": "H100 80GB",
"hbm_per_gpu_gb": 80,
"gpus_for_weights": 18
},
{
"config": "H200 141GB",
"hbm_per_gpu_gb": 141,
"gpus_for_weights": 10
},
{
"config": "B200 192GB",
"hbm_per_gpu_gb": 192,
"gpus_for_weights": 8
},
{
"config": "GB300 288GB",
"hbm_per_gpu_gb": 288,
"gpus_for_weights": 5
}
]Read those as a floor, not a target. They count weights only: no KV cache, no activation buffers, no allocator fragmentation, and no room for a second concurrent request. They also assume a parallel split that divides evenly, which 93 layers and 896 experts do not always allow.
Published guidance sits well above the floor. As of August 2026 Moonshot recommends a supernode of 64 or more accelerators, and the SGLang cookbook ships an H100 configuration built from four 8-GPU nodes, 32 GPUs and 2,560 GB of aggregate memory, against a floor of 18 cards. That gap is not waste. It is KV cache, activation memory, and the headroom that lets the server batch many requests at once. Even the friendliest row, 5 GB300 class cards, describes a machine most providers do not rent as a single SKU.
The KV cache is the part that surprises people
Weights are a fixed cost. The KV (key value) cache is not: it grows with context length and again with every concurrent user. For ordinary attention the formula is bytes per token = 2 * layers * kv_heads * head_dim * bytes_per_element, and then you multiply by context length and by concurrency.
Here is a worked example, and it is only an example: 64 layers, 8 KV heads, head dimension 128, fp8. That gives 2 64 8 128 1 = 131,072 bytes, so 128 KiB per token.
The data behind this chart
[
{
"label": "8k context",
"kv_gib_per_user": 1
},
{
"label": "32k context",
"kv_gib_per_user": 4
},
{
"label": "128k context",
"kv_gib_per_user": 16
},
{
"label": "1M context",
"kv_gib_per_user": 128
}
]One user at 128k context costs 16 GiB. One user at the full million costs 128 GiB, which is more than any single card holds, for one conversation.
K3 does not use ordinary attention, and that last number is why. Its 93 layers are 69 KDA (Kimi Delta Attention) layers and 24 Gated MLA (multi-head latent attention) layers. KDA keeps a fixed size recurrent state instead of a cache that grows with every token, and MLA compresses key and value into one low rank latent vector, so the real per-token cost falls a long way below the worked example. Moonshot has not published the latent dimensions, so I will not put a per-user figure on K3 itself. Measure yours instead: start the server with a small --max-model-len, watch memory with nvidia-smi, then raise the limit until allocation fails.
The shape of the reasoning survives the next release. If a model advertises a million token context and says nothing about its attention design, assume the cache is the binding constraint until someone proves otherwise.
Tier 1: rent the cluster by the hour
This is the only tier that runs K3 itself. You do not buy the hardware. You rent it for the hours you need it and you stop it afterwards.
The data behind this chart
[
{
"label": "1 GPU, always on",
"gpu_hours": 720,
"usd_cost": "1,800"
},
{
"label": "8 GPUs, 4 hours a day",
"gpu_hours": 960,
"usd_cost": "2,400"
},
{
"label": "8 GPUs, always on",
"gpu_hours": 5760,
"usd_cost": "14,400"
},
{
"label": "32 GPUs, always on",
"gpu_hours": 23040,
"usd_cost": "57,600"
}
]The rate is an assumption, not a quote. On-demand list prices for datacentre accelerators sat roughly between 2 and 5 USD per GPU hour through 2026, and reserved capacity is cheaper. Take your provider's real number and redo the multiplication: GPUs times hours times rate. The point of the chart is the ratio. Bursting an 8 GPU node for four hours a day costs 2,400 USD a month, while leaving the SGLang sized 32 GPU configuration running costs 57,600 USD.
Both mainstream servers publish a launch command on the model card.
pip install vllm
vllm serve "moonshotai/Kimi-K3"pip install sglang
python3 -m sglang.launch_server --model-path "moonshotai/Kimi-K3" --host 0.0.0.0 --port 30000Neither bare command is what you run on a real cluster. Add the parallelism flags that match your hardware: SGLang takes --tp-size for tensor parallel and --ep-size for expert parallel, and the product of those has to equal the GPUs you actually have.
Check the server came up before you send real traffic:
curl http://127.0.0.1:30000/v1/modelsA healthy server answers with a JSON object listing the model id. Connection refused means the process is still loading weights or has already exited, so read the server log before you retry.
The common failure on day one is a runtime older than the model. K3 shipped with KDA and a new MoE layer that the stable vLLM and SGLang releases did not carry at launch, and the symptom is the server exiting during startup with a line of the form Model architectures [...] are not supported for now. No config change fixes that, because the code to run those layers is not in your build. Install the nightly the model card names, or wait for the release that includes it.
One cost note that catches people. The meter starts when the instance starts, not when the model is ready. A 1.5 TB download at 1 GB/s is about 25 minutes of cluster time before the first token. Stage the weights on a volume that outlives the instance, so the second run starts in minutes.
Tier 2: run a smaller model on one accelerator
You are not running K3 in this tier. Say that out loud before you start, because most "run K3 locally" threads end here without admitting it.
The fit rule is the same formula in miniature: parameters times bytes per weight, plus KV cache, plus about 2 GB of runtime overhead, has to sit under your VRAM. At 4-bit that is roughly half a byte per parameter, which gives comfortable pairings:
- 16 GB card: a 7B model at 4-bit with room for a long context
- 24 GB card: a 14B model at 4-bit
- 48 GB card: a 32B model at 4-bit
- 80 GB card: a 70B model at 4-bit, or a 30B class MoE at 8-bit
Ollama is the shortest path to a working server on a VPS with a GPU attached:
curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:14bollama run downloads the model on first use, then drops you at a prompt. A tag that does not exist returns Error: model "..." not found, so copy tags from the library page rather than typing them from memory. The full walkthrough, including the systemd unit and remote access, is in running Ollama on a VPS.
llama.cpp gives you more control over quantisation and offload:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j./build/bin/llama-server -m model.gguf -c 8192 -ngl 99 --host 0.0.0.0 --port 8080-ngl 99 asks for every layer on the GPU. Read the load log: it prints how many layers were offloaded. Layers that spill into system RAM run at RAM bandwidth instead of HBM bandwidth, so generation speed drops by an order of magnitude the moment the model stops fitting. The trade-offs between the two tools are covered in Ollama and llama.cpp side by side.
Tier 3: hosted API, self-hosted orchestration
The data behind this chart
[
{
"label": "Input, cache hit",
"usd_per_million_tokens": "0.30"
},
{
"label": "Input, cache miss",
"usd_per_million_tokens": "3.00"
},
{
"label": "Output",
"usd_per_million_tokens": "15.00"
}
]The endpoint is OpenAI compatible, so an existing client works after you change the base URL.
curl https://api.moonshot.ai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $MOONSHOT_API_KEY" \
-d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "Say hello."}]}'A working key returns a JSON object with a choices array. A 401 means the key is wrong or the Bearer prefix is missing. A model-not-found error usually means the id changed, because providers retire ids between checkpoints.
Now the break-even, using the assumed rental rate above. An always-on 8 GPU node costs 14,400 USD a month, and at 15.00 USD per million output tokens the same money buys about 960 million output tokens from the API. To win on cost you have to generate close to a billion output tokens a month, roughly 30 million a day, and keep the cluster busy the whole time, because idle GPUs bill at the same rate as busy ones. Prompt-heavy agent workloads move the line further away still: repeated context bills at the cache-hit rate of 0.30 USD per million rather than the cache-miss rate of 3.00 USD.
What you self-host in this tier is everything around the model: a gateway that holds the API key so it never reaches a client, request and response logs, retries, rate limits, and per-user budgets. That runs on a small VPS with no GPU at all. The same split applies to closed weights, where self-hosting Claude is not possible at the model level and orchestration is the only part you own.
Which serving stack belongs to which tier
vLLM and SGLang class servers belong to tier 1. They exist to serve many requests at once, with continuous batching and a paged KV cache, plus tensor and expert parallelism spread across several nodes. They assume datacentre accelerators and a fast interconnect between them. On a single consumer card they are heavier to install and give you little you would notice.
llama.cpp and Ollama belong to tier 2. They target one machine, GGUF quantisation, CPU offload when the model does not fit, and low concurrency. llama.cpp will technically load an enormous MoE by keeping most layers in system RAM, and for a 2.8T model that path measures in seconds per token. It proves the file parses. It is not a service you can put users on. The full comparison is in Ollama against vLLM, and it does not change with the model: the question is always whether you are serving many users on shared hardware or one user on your own.
The four numbers that outlive this checkpoint
- Total parameters times bytes per weight gives the memory floor. Nothing runs below it, and no quantisation trick moves it much once the release is already 4-bit.
- Active parameters give the throughput class. A 2.8T MoE with 104B active computes like a 104B model.
- KV cache per token, times context length, times concurrency, is the cost that keeps growing after you have paid for the weights.
- Tokens per second per dollar is the only number that picks a tier. Everything above is input to it.
Apply those four to any release and you get the right answer before you open a vendor guide. Then date every figure you write down. Prices and supported-architecture lists both moved inside the two weeks after K3 launched, and every number on this page is one published in July 2026.
FAQ
Can I run Kimi K3 on a single GPU?
No. The weights are about 1.4 TB at the MXFP4 precision Moonshot ships, and the largest single accelerator on sale holds 288 GB. An MoE model cannot stream its inactive experts from disk at usable speed, because the router may pick any expert on any token and a PCIe fetch takes far longer than the token budget allows. The smallest sensible K3 deployment is a multi-GPU node, and published recipes use 32 accelerators or more.
How much VRAM does Kimi K3 need?
Start at 1.4 TB for weights alone, which is 18 H100 80GB cards or 5 GB300 class cards. Then add KV cache and activation memory on top. As of August 2026 Moonshot recommends 64 or more accelerators, and the SGLang cookbook publishes a 32 GPU H100 configuration with 2,560 GB aggregate, so treat the weights figure as a floor rather than a requirement.
Does quantisation make Kimi K3 fit on one node?
Not usefully. The released checkpoint is already 4-bit with quantisation-aware training, so the easy saving has been taken. Halving again to 2-bit brings the weights to 0.7 TB, which is still more than double the largest card, and the accuracy cost of 2-bit has not been measured on this model.
Is renting GPUs cheaper than the Kimi K3 API?
Only at high and steady volume. At an assumed 2.50 USD per GPU hour, an always-on 8 GPU node costs 14,400 USD a month, and the same money buys about 960 million output tokens at the published rate of 15.00 USD per million. You also pay for idle hours, weight downloads and the person who keeps the cluster alive. Rent by the hour for bursts, and compare against your own measured token volume instead of a guess.
What does 104B active parameters mean for speed?
It means the arithmetic per token is that of a 104B model, so throughput lands in that class rather than the 2.8T class. It says nothing about memory: all 2.8T parameters stay resident, because the router can call any expert on any token. Use the active count to predict tokens per second, and the total count to size VRAM.