Wetin E Take to Self-Host Kimi K3?

Wetin e take to self-host Kimi K3

Self-hosting Kimi K3 mean say you need find space for 2.8 trillion parameters. Moonshot publish the open weights for MXFP4. E dey use about half byte for each weight, so the weights alone reach roughly 1.4 TB before you allocate even one token for cache. No accelerator wey dey sell today fit hold all of that by itself. K3 na multi-node model, and for one server, answer na no.

Na the final verdict be that. Everything wey follow na the arithmetic behind am, because na this arithmetic you go reuse for the next release. Several infrastructure vendors publish K3 deployment guides during the weeks after the announcement on 17 July 2026, and each guide assume say you already get cluster. This page start from the other side: how much e cost, wetin you fit run instead, and how you go know which of the two situations you dey in.

Total parameters and active parameters no be the same number

K3 na mixture of experts model. MoE (mixture of experts) dey split the network into many sub-networks, then router dey choose small number of dem for each token. The model card list 2.8T total parameters and 104B activated per token, from 896 routed experts, where 16 dey run for any token, across 93 layers.

The two parameter counts answer different questions. Mixing dem up na the commonest mistake for every "can I run this" discussion.

Active parameters na dem dey set compute cost. Each token dey pass through about 104B parameters. So, the throughput wey you suppose expect go resemble 104B dense model, no be 2.8T model. Na the main reason to build MoE.

Total parameters na dem dey set memory cost. Router fit choose any expert for any token, so every expert must dey resident before the first request arrive. You no fit keep 104B for VRAM and fetch the rest only when you need dem, because the fetch must finish within microseconds, while PCIe link dey move tens of gigabytes per second. People dey try am. Streaming experts from NVMe fit change model wey suppose emit dozens of tokens per second to one wey dey emit one token every few seconds.

So, compute cheap, but storage expensive. Size the hardware against 2.8T. Base your speed expectation on 104B.

Bytes per weight, and where the terabytes come from

Parameter count multiply by bytes per weight. For the weights, na be the complete formula.

[
  {
    "label": "bf16",
    "bytes_per_weight": 2,
    "weights_tb": 5.6
  },
  {
    "label": "fp8",
    "bytes_per_weight": 1,
    "weights_tb": 2.8
  },
  {
    "label": "4-bit (MXFP4, as shipped)",
    "bytes_per_weight": 0.5,
    "weights_tb": 1.4
  },
  {
    "label": "2-bit",
    "bytes_per_weight": 0.25,
    "weights_tb": 0.7
  }
]

K3 train with quantisation awareness and release with MXFP4 weights and MXFP8 activations, so the 4-bit row na the real one. The rows wey dey above am na for scale: for bf16, the same model go need 5.6 TB. MXFP4 still dey store one shared 8-bit scale for every block of 32 weights. This add about 6 percent, so the published repository dey closer to 1.5 TB than a clean 1.4 TB.

This one close the usual escape route. "Just quantise it" no go help here, because the released checkpoint already dey 4-bit. If you reduce am to 2-bit, the weights go be 0.7 TB, and accuracy go drop without anybody measuring the result for this checkpoint. You still go far pass any single card.

Kimi K3 dey need how many GPUs

[
  {
    "config": "H100 80GB",
    "hbm_per_gpu_gb": 80,
    "gpus_for_weights": 18
  },
  {
    "config": "H200 141GB",
    "hbm_per_gpu_gb": 141,
    "gpus_for_weights": 10
  },
  {
    "config": "B200 192GB",
    "hbm_per_gpu_gb": 192,
    "gpus_for_weights": 8
  },
  {
    "config": "GB300 288GB",
    "hbm_per_gpu_gb": 288,
    "gpus_for_weights": 5
  }
]

Make you see dem as minimum, no be target. Dem count weights only: no KV cache, no activation buffers, no allocator fragmentation, and no space for second concurrent request. Dem also assume say parallel split go divide evenly, but 93 layers and 896 experts no always allow that.

Published guidance dey well above the minimum. As of August 2026, Moonshot recommend supernode wey get 64 or more accelerators, and SGLang cookbook get H100 configuration wey use four 8-GPU nodes, 32 GPUs and 2,560 GB aggregate memory, compared with minimum of 18 cards. That difference no be waste. Na KV cache, activation memory, and extra capacity wey allow server batch plenty requests at once. Even the most favourable row, 5 GB300-class cards, dey describe machine wey most providers no dey rent as one SKU.

The KV cache na part wey dey surprise people

Weights na fixed cost. KV (key value) cache no be fixed: e dey grow as context length increase, and e grow again for every concurrent user. For ordinary attention, formula na bytes per token = 2 * layers * kv_heads * head_dim * bytes_per_element. After that, multiply am by context length and concurrency.

Na example be this, and na only example: 64 layers, 8 KV heads, head dimension 128, fp8. E give 2 64 8 128 1 = 131,072 bytes, wey be 128 KiB for each token.

[
  {
    "label": "8k context",
    "kv_gib_per_user": 1
  },
  {
    "label": "32k context",
    "kv_gib_per_user": 4
  },
  {
    "label": "128k context",
    "kv_gib_per_user": 16
  },
  {
    "label": "1M context",
    "kv_gib_per_user": 128
  }
]

One user for 128k context go cost 16 GiB. One user for the full million go cost 128 GiB. This pass wetin any single card fit hold, for just one conversation.

K3 no use ordinary attention, and na why that last number dey important. E get 93 layers: 69 KDA (Kimi Delta Attention) layers and 24 Gated MLA (multi-head latent attention) layers. KDA dey keep recurrent state wey get fixed size, instead of cache wey dey grow with every token. MLA dey compress key and value into one low rank latent vector. So the real cost per token dey far below the example we calculate. Moonshot never publish the latent dimensions, so I no go give per-user figure for K3 itself. Measure your own instead: start the server with small --max-model-len, monitor memory with nvidia-smi, then raise the limit until allocation fail.

The reasoning still apply for the next release. If model advertise one million token context but e no talk about the attention design, assume say cache na the binding constraint until person prove otherwise.

Tier 1: rent the cluster by the hour

Na only this tier dey run K3 itself. You no need buy the hardware. You rent am for the hours wey you need, then stop am afterwards.

[
  {
    "label": "1 GPU, always on",
    "gpu_hours": 720,
    "usd_cost": "1,800"
  },
  {
    "label": "8 GPUs, 4 hours a day",
    "gpu_hours": 960,
    "usd_cost": "2,400"
  },
  {
    "label": "8 GPUs, always on",
    "gpu_hours": 5760,
    "usd_cost": "14,400"
  },
  {
    "label": "32 GPUs, always on",
    "gpu_hours": 23040,
    "usd_cost": "57,600"
  }
]

The rate na assumption, e no be quote. On-demand list prices for datacentre accelerators dey roughly between 2 and 5 USD per GPU hour through 2026, and reserved capacity cheaper. Use your provider real number and do the multiplication again: GPUs times hours times rate. The main point of the chart na the ratio. If you burst an 8 GPU node for four hours every day, e go cost 2,400 USD per month. But if you leave the SGLang sized 32 GPU configuration running, e go cost 57,600 USD.

Both mainstream servers publish launch command for the model card.

pip install vllm
vllm serve "moonshotai/Kimi-K3"

pip install sglang
python3 -m sglang.launch_server --model-path "moonshotai/Kimi-K3" --host 0.0.0.0 --port 30000

You no fit run either bare command for real cluster. Add the parallelism flags wey match your hardware: SGLang dey use --tp-size for tensor parallel and --ep-size for expert parallel. The product of both values must equal the number of GPUs wey you actually get.

Check say the server don come up before you send real traffic:

curl http://127.0.0.1:30000/v1/models

Healthy server go answer with JSON object wey list the model id. Connection refused mean say the process still dey load weights or e don already exit. Read the server log before you retry.

The common problem for day one na runtime wey old pass the model. K3 ship with KDA and a new MoE layer wey the stable vLLM and SGLang releases no support when dem launch. The symptom na server wey dey exit during startup with a line like Model architectures [...] are not supported for now. No config change fit fix am, because the code wey dey run those layers no dey inside your build. Install the nightly release wey the model card name, or wait for the release wey include am.

One cost matter wey dey catch people. The meter start when the instance start, no be when the model ready. A 1.5 TB download at 1 GB/s na about 25 minutes of cluster time before the first token. Stage the weights for a volume wey go outlive the instance, so the second run fit start within minutes.

Tier 2: run smaller model for one accelerator

For this tier, you no dey run K3. Talk am clearly before you start, because most “run K3 locally” threads dey end for here without admitting am.

The fit rule still use the same formula for smaller scale: parameters times bytes per weight, plus KV cache, plus about 2 GB runtime overhead, must fit inside your VRAM. For 4-bit, na roughly half byte per parameter, so these pairings dey comfortable:

• 16 GB card: 7B model for 4-bit, with space for long context
• 24 GB card: 14B model for 4-bit
• 48 GB card: 32B model for 4-bit
• 80 GB card: 70B model for 4-bit, or 30B-class MoE for 8-bit

Ollama na the shortest way to get working server for VPS wey get GPU attached:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:14b

ollama run go download the model the first time you use am, then e go take you to prompt. If tag no exist, e go return Error: model "..." not found, so copy tags from the library page instead of typing dem from memory. The complete walkthrough, including the systemd unit and remote access, dey for running Ollama for VPS.

llama.cpp give you more control over quantisation and offload:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

./build/bin/llama-server -m model.gguf -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

-ngl 99 dey request make every layer run for GPU. Read the load log: e go show how many layers dem offload. Layers wey spill enter system RAM go run with RAM bandwidth instead of HBM bandwidth, so generation speed go drop by one order of magnitude as soon as the model no fit again. Ollama and llama.cpp side by side explain the trade-offs between both tools.

Tier 3: hosted API, self-hosted orchestration

[
  {
    "label": "Input, cache hit",
    "usd_per_million_tokens": "0.30"
  },
  {
    "label": "Input, cache miss",
    "usd_per_million_tokens": "3.00"
  },
  {
    "label": "Output",
    "usd_per_million_tokens": "15.00"
  }
]

The endpoint dey OpenAI compatible, so any existing client go work after you change the base URL.

curl https://api.moonshot.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MOONSHOT_API_KEY" \
  -d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "Say hello."}]}'

Valid key dey return a JSON object wey get choices array. If you see 401, e mean say the key wrong or Bearer prefix dey miss. Model-not-found error usually mean say the id don change, because providers dey retire ids between checkpoints.

Now make we calculate when the cost balance, using the rental rate wey we assume above. An 8 GPU node wey dey always on cost 14,400 USD every month. For 15.00 USD per million output tokens, that same money fit buy about 960 million output tokens from the API. To save money, you need generate almost one billion output tokens every month, roughly 30 million every day, and keep the cluster busy throughout, because idle GPUs still cost the same rate as busy ones. Agent workloads wey use plenty prompts go push this break-even point further: repeated context dey cost 0.30 USD per million when cache hit happen, instead of 3.00 USD when cache miss happen.

For this tier, wetin you self-host na everything around the model: gateway wey keep the API key so e no reach any client, request and response logs, retries, rate limits, and per-user budgets. All this fit run for small VPS wey no get GPU. The same separation apply to closed weights, where self-hosting Claude no possible for the model level, and orchestration na the only part wey you control.

Which serving stack belong to which tier

vLLM and SGLang class servers belong to tier 1. Dem dey serve plenty requests at once, with continuous batching and paged KV cache, plus tensor and expert parallelism wey spread across several nodes. Dem assume say datacentre accelerators and fast interconnect dey between dem. For one consumer card, dem heavier to install, and you no go notice much benefit.

llama.cpp and Ollama belong to tier 2. Dem target one machine, GGUF quantisation, CPU offload when model no fit, and low concurrency. llama.cpp fit technically load one enormous MoE by keeping most layers for system RAM, but for 2.8T model, that method fit take seconds per token. E only prove say the file parse correctly. E no be service wey you fit put users on. Full comparison dey for Ollama against vLLM, and model no change am: the question always be whether you dey serve plenty users for shared hardware or one user for your own.

Numbers wey go still matter after this checkpoint

1. Total parameters multiply bytes per weight na the minimum memory wey model need. Nothing fit run below am, and no quantisation trick fit reduce am much once release don already be 4-bit.
2. Active parameters dey show the throughput class. 2.8T MoE wey get 104B active parameters dey calculate like 104B model.
3. KV cache per token, multiply context length, multiply concurrency, na the cost wey dey continue grow after you don pay for the weights.
4. Tokens per second per dollar na the only number wey fit choose tier. Everything wey dey above na input to am.

Apply these four to any release and you go get correct answer before you open vendor guide. Then put date for every figure wey you write down. Prices and supported-architecture lists both change within the two weeks after K3 launch, and every number for this page na one wey dem publish for July 2026.

FAQ

I fit run Kimi K3 for one GPU?

No. The weights na about 1.4 TB for the MXFP4 precision wey Moonshot release, and the biggest single accelerator wey dey sell fit hold 288 GB. MoE model no fit stream inactive experts from disk for usable speed, because router fit pick any expert for any token, and PCIe fetch dey take much longer than the token budget allow. The smallest K3 deployment wey make sense na node with plenty GPUs, and published recipes dey use 32 accelerators or more.

How much VRAM Kimi K3 need?

Start with 1.4 TB for weights alone. This na 18 H100 80GB cards or 5 GB300 class cards. Then add KV cache and activation memory on top. As of August 2026, Moonshot recommend 64 accelerators or more. SGLang cookbook publish 32 GPU H100 configuration with 2,560 GB aggregate, so treat the weights figure as minimum, no be exact requirement.

Quantisation fit make Kimi K3 enter one node?

No, e no really help. The checkpoint wey dem release don already use 4-bit with quantisation-aware training, so the easy memory saving don already happen. If you halve am again to 2-bit, the weights go become 0.7 TB. This still pass double the capacity of the biggest card, and nobody don measure the accuracy cost of 2-bit for this model.

Renting GPUs cheaper pass Kimi K3 API?

Only when volume high and steady. If we assume 2.50 USD per GPU hour, one 8 GPU node wey dey always on go cost 14,400 USD per month. The same money fit buy about 960 million output tokens for the published rate of 15.00 USD per million. You still go pay for idle hours, weight downloads, and the person wey dey keep the cluster alive. Rent by the hour for bursts, and compare am with your own measured token volume instead of guesswork.

Wetin 104B active parameters mean for speed?

E mean say the arithmetic per token na like 104B model, so throughput go dey for that class, no be 2.8T class. E no tell you anything about memory: all 2.8T parameters remain resident, because router fit call any expert for any token. Use active count to predict tokens per second, and total count to size VRAM.