SSD Nodes Learn Hosting plans →
How to do am Matt ConnorBy Matt Connor · Updated 2026-10-30

How I Fit Self-Host Kimi K3 Without 32 GPUs

Kimi K3 get 2.8T parameters and 104B active per token. See the VRAM arithmetic, KV cache math, and three honest ways to run am without 32 GPUs.

Wetin e take to self-host Kimi K3

Self-hosting Kimi K3 mean say you need find space for 2.8 trillion parameters. Moonshot publish the open weights for MXFP4. E dey use about half byte for each weight, so the weights alone reach roughly 1.4 TB before you allocate even one token of cache. No accelerator wey dey sell today fit hold all that by itself. For one GPU, the answer na no. For one server, the answer na yes for two ways. The first one na 8-GPU node with 288 GB accelerators. Na this setup the smallest published multi-user serving recipes dey use. The second one na one machine with about 1.6 TB combined RAM and VRAM. E go run Unsloth's lossless Q8 GGUF for llama.cpp, with or without GPU, and serve one user at a time.

Na the verdict be that. Everything below na the arithmetic behind am, because na this arithmetic you fit reuse for the next release. Several infrastructure vendors publish K3 deployment guides within the weeks after the announcement on 16 July 2026. Every guide assume say you already get cluster. This page start from the other side: wetin e cost, wetin you fit run instead, and how you go know which of the two situations you dey.

Total parameters and active parameters no be the same number

K3 na mixture of experts model. MoE (mixture of experts) dey split the network into many sub-networks, then router dey choose small number of dem for each token. The model card list 2.8T total parameters and 104B activated per token, from 896 routed experts. Na 16 experts dey activate for any particular token, across 93 layers.

Those two parameter counts dey answer different questions, and people dey swap dem often for every “can I run this” discussion.

Active parameters determine compute cost. A token dey multiply through about 104B parameters, so the throughput wey you suppose expect go resemble 104B dense model, no be 2.8T model. Na the main reason people dey build MoE.

Total parameters determine memory cost. Router fit select any expert for any token, so every expert must dey stored for fast storage before the first request arrive. E dey possible to keep only hot experts for VRAM and fetch the rest when needed: llama.cpp `--moe-cache-mib` dey keep GPU cache of experts wey otherwise dey system RAM, then upload the missing ones as router select dem. But this no reduce the total size. The remaining experts still need space for RAM, and PCIe link dey move tens of gigabytes per second compared with the terabytes per second wey HBM fit move, so every cache miss dey add delay. Streaming experts from NVMe because the model no fit inside RAM dey even slower, and Unsloth K3 guide warn say disk offloading dey run much slower.

So compute cost low, but storage cost high. Size the hardware for 2.8T. Base your speed expectation on 104B.

Bytes per weight, and where the terabytes come from

Parameter count multiply by bytes per weight. For the weights, na the complete formula be that.

ChartWeight footprint of 2.8 trillion parameters, by precision
Di data wey dey behind dis chart
[
  {
    "label": "bf16",
    "bytes_per_weight": 2,
    "weights_tb": 5.6
  },
  {
    "label": "fp8",
    "bytes_per_weight": 1,
    "weights_tb": 2.8
  },
  {
    "label": "4-bit (MXFP4, as shipped)",
    "bytes_per_weight": 0.5,
    "weights_tb": 1.4
  },
  {
    "label": "2-bit",
    "bytes_per_weight": 0.25,
    "weights_tb": 0.7
  }
]

K3 train with quantisation awareness and dem release am with MXFP4 weights and MXFP8 activations, so na the 4-bit row be the real one. The rows above am there to show scale: for bf16, the same model for want 5.6 TB. MXFP4 still dey store one shared 8-bit scale for every block of 32 weights, and this add about 6 percent. Moonshot config also leave attention, the shared experts and the vision tower outside the 4-bit format. So the complete checkpoint reach about 1.56 TB by Unsloth count, instead of a clean 1.4 TB.

This reduce the usual way to save space, but e no block am completely. The released checkpoint already dey 4-bit, so dem don use the easy space saving. But if dem go lower, the number still go reduce. As of 27 October 2026, the dynamic GGUF quantisations of K3 wey Unsloth measure run from 861.3 GB for the 2-bit UD-Q2_K_XL down to 594 GB for the 1-bit UD-IQ1_S. Unsloth publish the cost of this reduction: top-1 token agreement with the full model na 90.390 percent for UD-Q2_K_XL, 84.127 percent for the 711.1 GB UD-IQ2_XXS, and 78.875 percent for UD-IQ1_S. The Hugging Face repo also get smaller 1-bit files, down to 466 GB for UD-Q1_0, but e no publish agreement figure beside dem. This na measured accuracy loss. Even the smallest file still be about 1.6 times the biggest single card.

Kimi K3 need how many GPUs

ChartGPUs required to hold 1.4 TB of weights, before any KV cache
Di data wey dey behind dis chart
[
  {
    "config": "H100 80GB",
    "hbm_per_gpu_gb": 80,
    "gpus_for_weights": 18
  },
  {
    "config": "H200 141GB",
    "hbm_per_gpu_gb": 141,
    "gpus_for_weights": 10
  },
  {
    "config": "B200 192GB",
    "hbm_per_gpu_gb": 192,
    "gpus_for_weights": 8
  },
  {
    "config": "GB300 288GB",
    "hbm_per_gpu_gb": 288,
    "gpus_for_weights": 5
  }
]

Make una see these numbers as minimum, no be target. Dem count weights only: no KV cache, no activation buffers, no allocator fragmentation, and no extra space for second concurrent request. The split no need divide evenly: vLLM pipeline parallelism dey split model along layers and e support uneven splits, while SGLang K3 prefill recipes dey run 8 and 16 pipeline stages across 93 layers.

Published guidance dey above the minimum. Moonshot launch guidance recommend supernode wey get 64 or more accelerators. As of 27 October 2026, SGLang cookbook get H100 configuration wey use four 8-GPU nodes, 32 GPUs and 2,560 GB aggregate memory, compared with minimum of 18 cards. E also list single-node recipes for one 8-GPU B300 node and one 8-GPU MI350X or MI355X node, with 2,304 GB at 288 GB per card. The extra capacity above the minimum no be waste. Na for KV cache, activation memory, and headroom wey allow the server batch many requests at the same time. The friendliest row, 5 GB300-class cards, go round up for practical use to exactly that kind of 8-GPU node. Na this be the smallest machine wey anybody publish multi-user K3 serving recipe for.

KV cache na the part wey dey surprise people

Weights na fixed cost. KV (key value) cache no be fixed: e dey grow as context length increase, and e grow again with every concurrent user. For ordinary attention, formula na bytes per token = 2 * layers * kv_heads * head_dim * bytes_per_element, then you multiply am by context length and concurrency.

Make we use one worked example, but na example only: 64 layers, 8 KV heads, head dimension 128, fp8. This gives 2 64 8 128 1 = 131,072 bytes, so na 128 KiB for each token.

ChartKV cache per user in the worked example, at 128 KiB per token
Di data wey dey behind dis chart
[
  {
    "label": "8k context",
    "kv_gib_per_user": 1
  },
  {
    "label": "32k context",
    "kv_gib_per_user": 4
  },
  {
    "label": "128k context",
    "kv_gib_per_user": 16
  },
  {
    "label": "1M context",
    "kv_gib_per_user": 128
  }
]

One user for 128k context costs 16 GiB. One user for the full million costs 128 GiB. This nearly use half of a 288 GB card for one conversation, before you load even one weight.

K3 no use ordinary attention, and na why that last number matter. E get 93 layers: 69 KDA (Kimi Delta Attention) layers and 24 Gated MLA (multi-head latent attention) layers. KDA dey keep recurrent state wey get fixed size instead of cache wey dey grow with every token. MLA compresses key and value into one low rank latent vector, so the real cost for each token dey far below the worked example. Moonshot published config.json gives the size of that latent: a kv_lora_rank of 512 plus a 64-wide rotary key (qk_rope_head_dim), so na 576 values for each token in each MLA layer. Across 24 layers at 2 bytes each, this na 27,648 bytes, about 27 KiB for each token, or roughly 27 GiB for a full 1,048,576-token conversation, plus the fixed KDA state. Treat this as estimate, no be measurement: engines dey pad and arrange the cache in their own ways, and an 8-bit cache roughly reduce the size by half.

To see wetin your own deployment dey hold, no monitor nvidia-smi while you dey increase --max-model-len. vLLM preallocates its KV cache pool. E sizes the pool from --gpu-memory-utilization (0.92 by default for the current docs), or from explicit per-GPU cache budget. So total GPU memory shows the pool, no be one conversation. Read the startup log instead. There, vLLM reports the KV cache size in tokens and the maximum concurrency for your context length.

The main reasoning still apply for the next release. If model advertise one million token context and e no explain its attention design, assume say the cache na the binding constraint until somebody prove otherwise.

Tier 1: rent di cluster by di hour

Na this tier dey serve K3 for e shipped precision to plenty users at once. One machine wey get enough RAM fit also hold K3 for e released precision through Unsloth's Q8 GGUF, wey Unsloth label as full precision lossless, but that route dey serve one user at a time and tier 2 cover am. For here, you no dey buy the hardware. You rent am for di hours wey you need, then stop am afterwards.

ChartMonthly cost at an assumed 2.50 USD per GPU hour, 30 day month
Di data wey dey behind dis chart
[
  {
    "label": "1 GPU, always on",
    "gpu_hours": 720,
    "usd_cost": "1,800"
  },
  {
    "label": "8 GPUs, 4 hours a day",
    "gpu_hours": 960,
    "usd_cost": "2,400"
  },
  {
    "label": "8 GPUs, always on",
    "gpu_hours": 5760,
    "usd_cost": "14,400"
  },
  {
    "label": "32 GPUs, always on",
    "gpu_hours": 23040,
    "usd_cost": "57,600"
  }
]

Di rate na assumption, no be quote. On-demand rates dey vary plenty between providers and GPU generations. Di 288 GB parts wey di single-node recipes need cost more per hour than H100, while reserved capacity cheaper. Use your provider real number and redo di multiplication: GPUs times hours times rate. Di chart point na di ratio. Bursting an 8 GPU node for four hours every day cost 2,400 USD per month, while leaving di SGLang-sized 32 GPU H100 configuration running cost 57,600 USD.

Both mainstream servers publish launch command for di model card.

pip install vllm
vllm serve "moonshotai/Kimi-K3"
pip install sglang
python3 -m sglang.launch_server --model-path "moonshotai/Kimi-K3" --host 0.0.0.0 --port 30000

Neither of di bare commands na wetin you go run for real cluster. Add di parallelism flags wey match your hardware: SGLang dey use --tp-size for tensor parallel, --pp-size for pipeline parallel, and --ep-size for expert parallel. Tensor parallel times pipeline parallel must equal di number of GPUs wey you actually get. Expert parallel dey split di experts inside that group, na why di 32 GPU H100 recipe set both --tp-size 32 and --ep-size 32.

Check say di server don come up before you send real traffic:

curl http://127.0.0.1:30000/v1/models

Healthy server go answer with JSON object wey list di model id. Connection refused mean say di process still dey load weights or e don exit already, so read di server log before you retry.

Di common failure for di first weeks na runtime wey old pass di model. K3 ship with KDA and new MoE layer wey stable vLLM and SGLang releases no carry for launch. Di symptom na server wey dey exit during startup with line of di form Model architectures [...] are not supported for now. No config change fit fix am, because di code wey go run those layers no dey inside your build. This one don change since then: as of 27 October 2026, SGLang don carry K3 support for e releases since v0.5.17, with v0.5.22 as di latest, and vLLM v0.28.0 to v0.32.0 release notes all include K3 work. Install current release. Some hardware-specific paths still dey name dedicated image, for example SGLang NVFP4 builds on Blackwell, so check di recipe for your card.

One cost matter wey dey catch people. Di meter start when di instance start, no be when di model ready. Downloading 1.56 TB at 1 GB/s na about 26 minutes of cluster time before di first token. Stage di weights on volume wey go outlive di instance, so di second run go start within minutes.

Tier 2: run smaller model for one accelerator

For this tier, you no dey run K3. Talk am clearly before you start, because most “run K3 locally” threads dey end for here without admitting am.

The fit rule still be the same formula for smaller scale: parameters times bytes per weight, plus KV cache, plus about 2 GB runtime overhead, all must fit inside your VRAM. For 4-bit, na roughly half byte per parameter, so these pairings dey comfortable:

  • 16 GB card: 7B model for 4-bit, with space for long context
  • 24 GB card: 14B model for 4-bit
  • 48 GB card: 32B model for 4-bit
  • 80 GB card: 70B model for 4-bit, or 30B-class MoE for 8-bit

Every pairing above assume say only one request dey run at once. Once second person send prompt, every concurrent slot need its own KV cache. Ollama's NUM_PARALLEL and MAX_QUEUE settings handle the trade-off between parallel slots, queued requests, and the VRAM wey remain.

One way dey to run K3 itself for one large machine, and e dey between this tier and the last one. Unsloth documents GGUF builds of K3 for llama.cpp, from 1-bit and 2-bit quantisations up to UD-Q8_K_XL. E labels UD-Q8_K_XL as full precision lossless, and names NVIDIA DGX Station as hardware wey fit run dem. The rule of thumb be say RAM plus VRAM should roughly equal the quant size: from about 610 GB for the 1-bit UD-IQ1_S build to about 1.6 TB for Q8. If memory no reach, the model falls back to disk and e slow down sharply. GPU no compulsory: Unsloth build instructions switch CUDA off “if you don't have a GPU or just want CPU inference”. The 1-bit and 2-bit builds na K3 heavily quantised, so accuracy cost dey apply as measured above. The Q8 build na K3 for the precision wey e release with, served to one user at RAM speed. Neither one be smaller model.

Ollama na the shortest way to get working server for VPS wey get GPU attached:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:14b

ollama run go download the model the first time you use am, then e go take you to prompt. If tag no exist, e go return Error: model "..." not found, so copy tags from the library page instead of typing dem from memory. The full walkthrough, including the systemd unit and remote access, dey for running Ollama on VPS.

llama.cpp give you more control over quantisation and offload:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-server -m model.gguf -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

-ngl 99 dey request make every layer dey on GPU. Check the load log: e go show how many layers get offloaded. Layers wey spill enter system RAM dey run with RAM bandwidth instead of HBM bandwidth, so generation speed go drop by one order of magnitude as soon as the model no fit again. The trade-offs between both tools dey covered for Ollama and llama.cpp side by side.

Tier 3: hosted API, self-hosted orchestration

ChartKimi K3 published API pricing, USD per million tokens, checked 17 July 2026
Di data wey dey behind dis chart
[
  {
    "label": "Input, cache hit",
    "usd_per_million_tokens": "0.30"
  },
  {
    "label": "Input, cache miss",
    "usd_per_million_tokens": "3.00"
  },
  {
    "label": "Output",
    "usd_per_million_tokens": "15.00"
  }
]

The endpoint dey OpenAI compatible, so existing client go work after you change the base URL.

curl https://api.moonshot.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MOONSHOT_API_KEY" \
  -d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "Say hello."}]}'

Working key go return JSON object wey get choices array. If e return 401, authentication fail: check say the key correct, e never expire or dem never disable am, and say the header read Authorization: Bearer followed by the key. If e show model-not-found error, e usually mean say the id don change, because providers dey retire ids between checkpoints.

Now make we calculate break-even with the rental rate we assume above. Always-on 8 GPU node cost 14,400 USD every month. At 15.00 USD per million output tokens, that same money fit buy about 960 million output tokens from the API. To save money, you need generate almost 1 billion output tokens every month, roughly 30 million every day, and keep the cluster busy throughout. Idle GPUs still dey charge at the same rate as busy ones. Prompt-heavy agent workloads move the break-even point further away: repeated context dey charge at the cache-hit rate of 0.30 USD per million instead of the cache-miss rate of 3.00 USD.

For this tier, wetin you self-host na everything around the model: gateway wey hold the API key so e no ever reach client, request and response logs, retries, rate limits, and per-user budgets. That one fit run for small VPS without any GPU. The same split apply to closed weights, where self-hosting Claude no dey possible for model level, and orchestration na the only part wey belong to you.

Which serving stack belong to which tier

vLLM and SGLang class servers dey for tier 1. Dem dey made to serve plenty requests at the same time, with continuous batching and paged KV cache, plus tensor and expert parallelism wey spread across several nodes. Dem assume say datacentre accelerators and fast interconnect dey between dem. For one consumer card, dem harder to install and no give you much benefit wey you go notice.

llama.cpp and Ollama dey for tier 2. Dem target one machine, GGUF quantisation, CPU offload when model no fit enter, and low concurrency. llama.cpp fit load very big MoE by keeping expert weights for system RAM (--cpu-moe or --n-cpu-moe), and with --moe-cache-mib, e fit cache the experts wey e dey use for GPU. Na so Unsloth K3 GGUF builds fit run for one machine. Speed then depend on RAM bandwidth and how often router miss the cache. If file big pass RAM plus VRAM, e go fall back to disk. E workable for one user. E no be service wey you fit put plenty users on. Full comparison dey for Ollama against vLLM, and model no change this matter: question always be whether you dey serve plenty users on shared hardware or one user for your own machine.

Numbers four wey go outlive this checkpoint

  1. Total parameters times bytes per weight na the memory floor. Nothing fit run below am for the precision wey dem release am with. Quantising below a 4-bit release fit save up to about 70 percent, but e go cost accuracy; e no be order-of-magnitude gain.
  2. Active parameters dey show the throughput class. A 2.8T MoE wey get 104B active parameters dey compute like 104B model.
  3. KV cache per token, times context length, times concurrency, na the cost wey continue to grow after you don pay for the weights.
  4. Tokens per second per dollar na the only number wey fit choose a tier. Everything above na input to am.

Apply these four to any release and you go get the correct answer before you open vendor guide. The next one don already get announcement: Moonshot rolled out K3.1 Pro, wey dem build on K3 with more post-training, inside Kimi Code on 23 October 2026. The API quickstart talk say dem go release the full weights by 4 November. So, always put date for every figure wey you write down. Prices and supported-architecture lists both change inside the two weeks after K3 launch. The launch figures for this page dem publish for July 2026. Dem rechecked the serving recipes, runtime releases, and quantisation numbers on 27 October 2026.

FAQ

I fit run Kimi K3 for one GPU?

No. The weights na about 1.4 TB for the MXFP4 precision wey Moonshot ships, and even the smallest published quantisation of K3, Unsloth's 466 GB UD-Q1_0 file, na about 1.6 times the 288 GB wey the biggest widely available accelerators get. llama.cpp fit keep experts for system RAM and cache the active ones for GPU, but e still need roughly the quant size for RAM plus VRAM. If memory no reach that level, the model go use disk and run much slower. The smallest published serving recipes use one 8-GPU node with B300 or MI350X/MI355X cards, while the H100 recipe use 32.

How much VRAM Kimi K3 need?

Start with 1.4 TB for weights alone. That na 18 H100 80GB cards or 5 GB300 class cards. Then add KV cache and activation memory on top. Moonshot launch guidance recommend 64 or more accelerators. As of 27 October 2026, the SGLang cookbook publish single-node recipes with 8 B300 or 8 MI350X/MI355X cards, 2,304 GB aggregate, plus a 32 GPU H100 configuration with 2,560 GB aggregate. So treat the weights figure as the minimum, not the full requirement.

Quantisation fit make Kimi K3 fit for one node?

The shipped 4-bit checkpoint already fit for one node if that node get eight 288 GB cards. Na this setup SGLang B300 and MI355X recipes use. E also fit for one machine with about 1.6 TB RAM plus VRAM through Unsloth's lossless Q8 GGUF. Lower quantisation help smaller machines. Unsloth measured dynamic GGUF builds range from 861.3 GB at 2-bit to 594 GB at 1-bit, and the repo list 1-bit files down to 466 GB, compared with about 1.56 TB lossless. Dem don measure the accuracy cost: top-1 agreement with the full model drop to 90.390 percent at UD-Q2_K_XL and 78.875 percent at UD-IQ1_S. None of dem fit on one card.

Renting GPUs cheaper pass Kimi K3 API?

Only when volume high and steady. With an assumed 2.50 USD per GPU hour, an always-on 8 GPU node cost 14,400 USD every month. That same money buy about 960 million output tokens at the published rate of 15.00 USD per million. You still pay for idle hours, weight downloads, and the person wey maintain the cluster. Rent by the hour for bursts, and compare am with your own measured token volume instead of guesswork.

Wetin 104B active parameters mean for speed?

E mean say the arithmetic per token na for a 104B model, so throughput go land for the 104B-model class instead of the 2.8T class. E no tell you anything about memory: all 2.8T parameters remain resident, because the router fit call any expert for any token. Use the active count to predict tokens per second, and use the total count to size VRAM.

Dis article help you?