How I Fit Self-Host Kimi K3 Without 32 GPUs

Wetín e take to self-host Kimi K3

Self-hosting Kimi K3 mean say you need find space for 2.8 trillion parameters. Moonshot publish the open weights for MXFP4, wey na about half byte for each weight, so the weights alone dey reach roughly 1.4 TB before you allocate even one token of cache. No accelerator wey dey for sale today fit hold all that by itself. K3 na multi-node model, and for one server, answer na no.

Na that be the verdict. Everything wey dey below na the arithmetic behind am, because na this arithmetic you fit reuse for the next release. Several infrastructure vendors publish K3 deployment guides for the weeks after the announcement on 17 July 2026, and each guide assume say you don already get cluster. This page start from the other side: how much e go cost, wetin you fit run instead, and how you fit know which of the two situation you dey.

Total parameters and active parameters no be the same number

K3 na mixture of experts model. MoE (mixture of experts) dey split network into many sub-networks, then router go select some of dem for each token. The model card list 2.8T total parameters and 104B activated per token, from 896 routed experts. Na 16 experts dey activate for any token, across 93 layers.

Those two parameter counts dey answer different questions. Mixing dem up na the most common mistake for every "can I run this" discussion.

Active parameters dey determine compute cost. Each token dey pass through about 104B parameters. So, the throughput wey you suppose expect go resemble 104B dense model, not 2.8T model. Na the main reason people dey build MoE.

Total parameters dey determine memory cost. Router fit select any expert for any token. So every expert must dey resident before the first request arrive. You no fit keep 104B for VRAM and fetch the remaining ones when needed. The fetch must finish within microseconds, while PCIe link dey move tens of gigabytes per second. People dey try am. Streaming experts from NVMe fit change model wey suppose emit dozens of tokens per second into one wey dey emit one token every few seconds.

So compute am cheap, but store am expensive. Size the hardware based on 2.8T. Base your speed expectation on 104B.

Bytes per weight, and where the terabytes come from

Parameter count multiply by bytes per weight. For weights, na be this whole formula.

[
  {
    "label": "bf16",
    "bytes_per_weight": 2,
    "weights_tb": 5.6
  },
  {
    "label": "fp8",
    "bytes_per_weight": 1,
    "weights_tb": 2.8
  },
  {
    "label": "4-bit (MXFP4, as shipped)",
    "bytes_per_weight": 0.5,
    "weights_tb": 1.4
  },
  {
    "label": "2-bit",
    "bytes_per_weight": 0.25,
    "weights_tb": 0.7
  }
]

K3 train with quantisation awareness and dem release am with MXFP4 weights and MXFP8 activations, so the 4-bit row na the real one. The rows wey dey above am na for scale: for bf16, the same model for need 5.6 TB. MXFP4 still dey store one shared 8-bit scale for every block of 32 weights, and this add about 6 percent. Na why the published repository dey nearer 1.5 TB than clean 1.4 TB.

This one close the usual escape hatch. "Just quantise it" no go help here, because the released checkpoint already be 4-bit. If you reduce am to 2-bit, the weights go become 0.7 TB, and accuracy go cost something wey nobody don measure for this checkpoint. You still go dey far pass wetin any single card fit handle.

How many GPUs Kimi K3 need

[
  {
    "config": "H100 80GB",
    "hbm_per_gpu_gb": 80,
    "gpus_for_weights": 18
  },
  {
    "config": "H200 141GB",
    "hbm_per_gpu_gb": 141,
    "gpus_for_weights": 10
  },
  {
    "config": "B200 192GB",
    "hbm_per_gpu_gb": 192,
    "gpus_for_weights": 8
  },
  {
    "config": "GB300 288GB",
    "hbm_per_gpu_gb": 288,
    "gpus_for_weights": 5
  }
]

Treat dem as minimum, no be target. Dem only count weights: no KV cache, no activation buffers, no allocator fragmentation, and no extra space for second concurrent request. Dem still assume say parallel split go divide evenly, but 93 layers and 896 experts no always allow that.

Published guidance dey well above the minimum. As of August 2026, Moonshot recommend supernode wey get 64 or more accelerators, and SGLang cookbook get H100 configuration wey use four 8-GPU nodes, 32 GPUs, and 2,560 GB aggregate memory, against minimum of 18 cards. That difference no be waste. Na KV cache, activation memory, and enough headroom make the server batch many requests at once. Even the most favourable row, 5 GB300-class cards, describe machine wey most providers no dey rent as one SKU.

KV cache na the part wey dey surprise people

Weights na fixed cost. KV (key value) cache no be fixed: e dey grow as context length increase, and e grow again for every concurrent user. For ordinary attention, formula na bytes per token = 2 * layers * kv_heads * head_dim * bytes_per_element. After that, multiply am by context length and concurrency.

See one worked example. Na only example be this: 64 layers, 8 KV heads, head dimension 128, fp8. E give 2 64 8 128 1 = 131,072 bytes, so na 128 KiB for each token.

[
  {
    "label": "8k context",
    "kv_gib_per_user": 1
  },
  {
    "label": "32k context",
    "kv_gib_per_user": 4
  },
  {
    "label": "128k context",
    "kv_gib_per_user": 16
  },
  {
    "label": "1M context",
    "kv_gib_per_user": 128
  }
]

One user for 128k context cost 16 GiB. One user for the full million cost 128 GiB. This pass wetin any single card fit hold, even for just one conversation.

K3 no use ordinary attention, and na why that last number matter. E get 93 layers: 69 KDA (Kimi Delta Attention) layers and 24 Gated MLA (multi-head latent attention) layers. KDA dey keep recurrent state wey get fixed size, instead of cache wey dey grow with every token. MLA compress key and value into one low rank latent vector. So the real cost per token dey far below the worked example. Moonshot never publish the latent dimensions, so I no go give per-user figure for K3 itself. Measure your own one instead: start the server with small --max-model-len, monitor memory with nvidia-smi, then increase the limit until allocation fail.

The reasoning still apply for the next release. If model advertise one million token context but no talk about its attention design, assume say cache na the binding constraint until person prove otherwise.

Tier 1: rent the cluster by the hour

Na only this tier dey run K3 itself. You no dey buy the hardware. You rent am for the hours wey you need, then stop am afterwards.

[
  {
    "label": "1 GPU, always on",
    "gpu_hours": 720,
    "usd_cost": "1,800"
  },
  {
    "label": "8 GPUs, 4 hours a day",
    "gpu_hours": 960,
    "usd_cost": "2,400"
  },
  {
    "label": "8 GPUs, always on",
    "gpu_hours": 5760,
    "usd_cost": "14,400"
  },
  {
    "label": "32 GPUs, always on",
    "gpu_hours": 23040,
    "usd_cost": "57,600"
  }
]

The rate na assumption, no be quotation. On-demand list prices for datacentre accelerators dey roughly between 2 and 5 USD per GPU hour through 2026, and reserved capacity dey cheaper. Use your provider real number and do the multiplication again: GPUs times hours times rate. The chart point na the ratio. If you burst an 8 GPU node for four hours every day, e go cost 2,400 USD per month, while leaving the SGLang-sized 32 GPU configuration running go cost 57,600 USD.

Both mainstream servers publish a launch command for the model card.

pip install vllm
vllm serve "moonshotai/Kimi-K3"

pip install sglang
python3 -m sglang.launch_server --model-path "moonshotai/Kimi-K3" --host 0.0.0.0 --port 30000

You no fit run either bare command for real cluster. Add the parallelism flags wey match your hardware: SGLang use --tp-size for tensor parallel and --ep-size for expert parallel, and the product of both must equal the GPUs wey you actually get.

Check say the server don come up before you send real traffic:

curl http://127.0.0.1:30000/v1/models

Healthy server dey answer with JSON object wey list the model id. Connection refused mean say process still dey load weights or e don already exit, so read the server log before you retry.

The common failure for day one na runtime wey old pass the model. K3 ship with KDA and new MoE layer wey stable vLLM and SGLang releases no carry when dem launch, and the symptom na server wey dey exit during startup with line of this form: Model architectures [...] are not supported for now. No config change fit fix am, because the code wey go run those layers no dey inside your build. Install the nightly wey model card name, or wait for the release wey include am.

One cost matter wey dey catch people. The meter start when instance start, no be when model ready. 1.5 TB download for 1 GB/s na about 25 minutes of cluster time before the first token. Stage the weights on volume wey go outlive the instance, so the second run fit start within minutes.

Tier 2: run smaller model for one accelerator

You no dey run K3 for this tier. Talk am clearly before you start, because most “run K3 locally” discussions dey end here without admitting am.

The fit rule na the same formula for smaller scale: parameters times bytes per weight, plus KV cache, plus about 2 GB runtime overhead, must fit inside your VRAM. For 4-bit, na roughly half byte per parameter, and this gives pairings wey get enough space:

• 16 GB card: 7B model for 4-bit, with space for long context
• 24 GB card: 14B model for 4-bit
• 48 GB card: 32B model for 4-bit
• 80 GB card: 70B model for 4-bit, or 30B class MoE for 8-bit

Every pairing above assume say na one request at a time. Once second person send prompt, every concurrent slot need its own KV cache. Na this trade Ollama NUM_PARALLEL and MAX_QUEUE settings dey help you manage between parallel slots, queued requests, and the VRAM wey remain.

Ollama na the shortest way to get working server for VPS wey get GPU attached:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:14b

ollama run go download the model the first time you use am, then e go take you enter prompt. If tag no exist, e go return Error: model "..." not found. So copy tags from library page instead of typing dem from memory. The complete walkthrough, including systemd unit and remote access, dey for running Ollama for VPS.

llama.cpp give you more control over quantisation and offload:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

./build/bin/llama-server -m model.gguf -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

-ngl 99 dey request make every layer dey on GPU. Read load log: e go show how many layers dem offload. Layers wey spill enter system RAM go run with RAM bandwidth instead of HBM bandwidth, so generation speed go drop by one order of magnitude as soon as model no fit again. The trade-offs between both tools dey covered for Ollama and llama.cpp side by side.

Tier 3: hosted API, self-hosted orchestration

[
  {
    "label": "Input, cache hit",
    "usd_per_million_tokens": "0.30"
  },
  {
    "label": "Input, cache miss",
    "usd_per_million_tokens": "3.00"
  },
  {
    "label": "Output",
    "usd_per_million_tokens": "15.00"
  }
]

The endpoint dey OpenAI compatible, so existing client go work after you change the base URL.

curl https://api.moonshot.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MOONSHOT_API_KEY" \
  -d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "Say hello."}]}'

Working key dey return JSON object wey get one choices array. 401 means say the key no correct or Bearer prefix dey miss. Model-not-found error usually mean say the id don change, because providers dey retire ids between checkpoints.

Now make we calculate the break-even point, using the rental rate we assume before. Always-on 8 GPU node cost 14,400 USD every month. For 15.00 USD per million output tokens, that same money fit buy about 960 million output tokens from the API. To save money, you need generate close to one billion output tokens every month, roughly 30 million per day, and keep the cluster busy throughout, because idle GPUs still cost the same rate as busy ones. Prompt-heavy agent workloads make the difference even bigger. Repeated context dey cost 0.30 USD per million at the cache-hit rate, instead of the cache-miss rate of 3.00 USD.

For this tier, wetin you self-host na everything around the model: gateway wey dey keep the API key so e no ever reach client, request and response logs, retries, rate limits, and per-user budgets. You fit run all of that for small VPS wey no get GPU at all. The same split apply to closed weights. For that case, self-hosting Claude no possible for the model level, so orchestration na the only part wey you own.

Which serving stack belong to which tier

vLLM and SGLang class servers belong to tier 1. Dem dey serve plenty requests at once, with continuous batching and paged KV cache, plus tensor and expert parallelism wey spread across several nodes. Dem assume say datacentre accelerators and fast interconnect dey between dem. For one consumer card, dem dey heavier to install and the benefit wey you go notice small.

llama.cpp and Ollama belong to tier 2. Dem target one machine, GGUF quantisation, CPU offload when model no fit enter, and low concurrency. llama.cpp fit technically load one enormous MoE by keeping most layers for system RAM, but for 2.8T model, that path dey take seconds per token. E prove say the file fit parse. E no be service wey you fit put users on. Full comparison dey for Ollama against vLLM, and model no change this: the question always be whether you dey serve plenty users on shared hardware or one user for your own.

The four numbers wey go pass this checkpoint

1. Total parameters times bytes per weight na the lowest memory wey you need. Nothing fit run below am, and no quantisation trick fit reduce am much once the release don already be 4-bit.
2. Active parameters dey show the throughput class. A 2.8T MoE wey get 104B active parameters dey compute like 104B model.
3. KV cache per token, times context length, times concurrency, na the cost wey dey continue grow after you don pay for the weights.
4. Tokens per second per dollar na the only number wey fit choose a tier. Everything wey dey above na input for am.

Apply these four to any release and you go get the correct answer before you open vendor guide. Then put date for every figure wey you write down. Prices and supported-architecture lists both change within the two weeks after K3 launch, and every number for this page na one wey dem publish for July 2026.

FAQ

I fit run Kimi K3 for one GPU?

No. The weights na about 1.4 TB for the MXFP4 precision wey Moonshot dey ship, and the biggest single accelerator wey dey sale fit hold 288 GB. MoE model no fit stream inactive experts from disk for speed wey useful, because router fit pick any expert for any token and PCIe fetch dey take far pass the token budget wey dey available. The smallest sensible K3 deployment na multi-GPU node, and published recipes dey use 32 accelerators or more.

How much VRAM Kimi K3 need?

Start with 1.4 TB for weights alone. This na 18 H100 80GB cards or 5 GB300 class cards. Then add KV cache and activation memory on top. As of August 2026, Moonshot dey recommend 64 or more accelerators. The SGLang cookbook also publish 32 GPU H100 configuration with 2,560 GB aggregate. So treat the weights figure as minimum, no be complete requirement.

Quantisation fit make Kimi K3 fit for one node?

No, e no go help enough. The released checkpoint already be 4-bit with quantisation-aware training, so the easy memory saving don already happen. If you halve am again to 2-bit, the weights go reach 0.7 TB. This still pass double the biggest card, and nobody don measure the accuracy cost of 2-bit for this model.

Renting GPUs cheaper than Kimi K3 API?

Only when volume high and steady. If we assume 2.50 USD per GPU hour, an always-on 8 GPU node go cost 14,400 USD per month. The same money fit buy about 960 million output tokens at the published rate of 15.00 USD per million. You still go pay for idle hours, weight downloads, and the person wey dey keep the cluster alive. Rent by the hour for bursts, and compare am with your own measured token volume instead of guesswork.

Wetin 104B active parameters mean for speed?

E mean say the arithmetic per token be like 104B model, so throughput go enter that class, no be the 2.8T class. E no tell you anything about memory: all 2.8T parameters remain resident, because router fit call any expert for any token. Use active count to predict tokens per second, and total count to size VRAM.