Can you run Inkling 2 Small on a VPS?
Inkling 2 Small does not run on an ordinary VPS. See the large memory CPU exception, Ollama Cloud route, GPU requirements and current API costs.
Can you run Inkling 2 Small on a VPS?
No, not on an ordinary VPS. The large-memory CPU exception needs roughly 180 GB or more of free RAM, Unsloth's roughly 1-bit UD-IQ1_S GGUF quantization and a llama.cpp build with Inkling 2 support. The RAM floor is arithmetic from 171.7 GB of weight files plus room for the KV cache, runtime and operating system, not a memory requirement published by Unsloth. A 594B MoE model on CPU is slow, and UD-IQ1_S is the lowest-quality quantization Unsloth offers. llama.cpp release b12406 (15 November 2026) and later include Inkling 2 support from PR #32217. Unsloth's guide still builds from open PR #32187; it does not confirm that its files load with the upstream release.
Two other guides may fit your goal better. To call a big model from a small VPS without loading any weights, read how to run Ollama Cloud models on a small VPS. To pick a model that really runs on your own VPS, start with Ollama vs llama.cpp for running models locally.
The full answer has three parts. First, Unsloth publishes GGUF quantizations with a CPU inference path for machines with enough RAM. The Hugging Face model card says the recommended NVFP4 checkpoint needs a GPU cluster with at least 500 GB of aggregated VRAM (video memory, the memory on the GPUs). That requirement applies to the official NVFP4 deployment, not every quantization. Second, a small VPS can still use the model through Ollama Cloud. The model then runs on Ollama's servers, which means your prompts leave your machine. Third, you can serve the open weights yourself on a GPU cluster you control, with vLLM or SGLang.
What is Inkling 2 Small?
Inkling 2 Small is an open-weight model from Thinking Machines Lab. The launch post calls it the lab's second model generation. The lab introduced it on 5 November 2026. An update to the same post, dated 12 November 2026, says the weights are now available on Hugging Face. These are the key facts as of 13 November 2026, taken from the launch post and the model card:
- Design: Mixture-of-Experts (MoE), with 594B total parameters and 17B active parameters per token.
- Layout: a 48-layer decoder-only transformer. Each token is routed to 8 of 384 experts.
- Input and output: text and images in, text out. The launch post says there is no audio support.
- Vision: a 292M-parameter vision encoder based on Perception Encoder.
- Reasoning: the effort level is variable. The lab ran its benchmarks at effort 0.99, which it calls xhigh.
- Context window: up to 1M tokens, according to the card.
- Licence: the card lists apache-2.0.
- Checkpoints: NVFP4 is the main one. MXFP8 is an alternative in a separate repo.
- Ollama: one tag,
inkling-2-small:cloud, and no tag that runs locally.
Hugging Face's own parameter counter now shows 594B for the NVFP4 repo, matching the lab's stated total. The repository includes the corrective commit "Add minimal quantization config to correct Hub parameter count (#3)".
The 12 November update also names the projects that support the model. For inference: Unsloth, SGLang, vLLM and TokenSpeed. For post-training: Miles, SkyRL and Snowflake Cortex Training. For hosted APIs: NVIDIA, Fireworks, Together, Baseten, Nebius, Modal and Cloudflare. Tinker, the lab's own service for training and sampling, offers it as well.
Why 17B active does not mean a 17B model
A dense model uses every parameter for every token. A Mixture-of-Experts model splits most of its feed-forward layers into many smaller networks, called experts. A small router picks a few experts for each token. In Inkling 2 Small the router picks 8 of 384 experts per token. So the compute for one token is close to that of a 17B dense model. That is all "17B active" means.
Memory is a separate question. The router picks again for every token, and the next token can need any of the 384 experts. So the full set of expert weights must be available to the runtime, in GPU memory or system RAM depending on the deployment. Memory and download size follow the 594B total, not the 17B active.
Here is the arithmetic, from the Hugging Face file listings and the model card on 13 November 2026:
The data behind this chart
[
{
"label": "NVFP4 download",
"gigabytes": 394
},
{
"label": "NVFP4 GPU memory floor",
"gigabytes": 500
},
{
"label": "MXFP8 download",
"gigabytes": 623
},
{
"label": "MXFP8 GPU memory floor",
"gigabytes": 800
}
]The NVFP4 repo is about 394 GB of files: 24 safetensors shards plus a 3.42 GB mtp.safetensors file. The MXFP8 repo shows 623 GB. Those figures are the weights alone. A running server also needs memory for the KV cache. That is the attention state it keeps for each request, and it grows with the context length. How the KV cache differs from prompt caching explains that memory in more detail.
Now compare that with a VPS. A typical VPS has no GPU at all, and its RAM is a few GB to a few dozen GB. A bigger disk alone would not help: 500 GB of aggregated VRAM is the model card's floor for the official NVFP4 deployment. Unsloth's GGUF builds instead support CPU inference, so a sufficiently large RAM VPS can load a quantization with room left for KV cache and runtime overhead. For where model files live on disk when a model does fit, see ollama pull vs run and where model files are stored. For a comparison with another MoE model walked through the same way, read what it takes to self-host Qwen 3.8 Flash Next.
Route 1: use Inkling 2 Small from a small VPS with Ollama Cloud
As of 13 November 2026, Ollama's library lists Inkling 2 Small with a single tag, inkling-2-small:cloud. The tag is labelled vision, tools, thinking and cloud. It takes text and image input and lists a 1M context. Ollama's cloud documentation says inference runs on Ollama's cloud service. Your VPS only sends the request and receives the reply, so it needs no GPU and very little memory.
With the Ollama CLI installed and signed in to your Ollama account on the VPS, this starts a chat:
ollama run inkling-2-small:cloudYou can also call the cloud API directly, with no Ollama install at all. Ollama's docs say both ways need an Ollama account. The direct API takes an API key in a bearer header. Watch the model name here: the cloud API uses inkling-2-small, with no :cloud suffix.
echo ${OLLAMA_API_KEY:+key is set}
curl https://ollama.com/api/chat \
-H "Authorization: Bearer $OLLAMA_API_KEY" \
-d '{
"model": "inkling-2-small",
"messages": [{"role": "user", "content": "Explain a Mixture-of-Experts model in two sentences."}],
"stream": false
}'The first line should print key is set. An empty line means the variable is not set in this shell, so the request would go out with an empty bearer token. Set the key before you send it. A healthy reply is one JSON object, and the model's answer is in its message.content field. The account, sign-in and key steps are in the guide to Ollama Cloud models on a small VPS, so they are not repeated here.
Ollama's library page also shows how to point a coding agent at the model:
ollama launch claude --model inkling-2-small:cloudThe same page lists opencode, hermes and openclaw as other targets for ollama launch. For how that command connects an agent to Ollama, read how to use Ollama with your coding agent.
What leaves your machine with Ollama Cloud
Every prompt, image and tool result you send goes to Ollama's servers. Ollama's pricing FAQ makes several statements about that data. It says: "Prompt or response data is never logged or trained on." It says the providers it partners with must have "no logging, no training, and zero data retention policies in place." It also says Ollama hosts models and compute primarily in the United States, and may route requests to Europe for extra capacity.
Those are promises from a provider. They are a different thing from local inference, where the prompt never leaves your hardware. If that difference matters for your data, read Ollama Cloud vs your own server before you send anything sensitive.
If the same VPS also runs an Ollama server for small local models, keep its API port, 11434, off the public internet. An open port lets anyone who finds it send requests to your server. How to secure your Ollama API endpoint explains how to lock it down.
What Ollama Cloud costs for this model
The data behind this chart
[
{
"label": "Ollama Cloud (13 Nov 2026)",
"input_usd": "0.30",
"cached_input_usd": "0.03",
"output_usd": "1.00"
},
{
"label": "Tinker sampling, 64K (19 Oct 2026)",
"input_usd": "0.30",
"cached_input_usd": "0.03",
"output_usd": "1.00"
},
{
"label": "Tinker sampling, 256K (19 Oct 2026)",
"input_usd": "0.30",
"cached_input_usd": "0.03",
"output_usd": "1.00"
},
{
"label": "Tinker serverless beta, 256K (19 Oct 2026)",
"input_usd": "0.30",
"cached_input_usd": "0.03",
"output_usd": "1.00"
}
]As of 13 November 2026, Ollama's pricing page lists inkling-2-small at $0.30 per million input tokens, $0.03 per million cached input tokens and $1.00 per million output tokens. The page offers off-peak rates only for deepseek-v4.1-flash and deepseek-v4-pro, so this model costs the same at any hour. Ollama measures usage in tokens at each model's rates and pays for it from credits. Pro, Max and Team plans include a monthly amount of credits. Ollama uses those first, before it draws on any credits you bought. For what each plan includes, see what is free in Ollama, locally and in the cloud.
Route 2: Tinker and the hosted APIs
Tinker is the lab's own service. Its pricing page marks its tables "as of October 19, 2026", and on 13 November 2026 it still showed the same figures. For Inkling 2 Small, sampling costs $0.30 per million input tokens, $0.03 cached and $1.00 output, at both the 64K and the 256K context. The serverless inference beta has the same three prices at 256K. The page says production use of serverless inference requires contacting the Tinker team.
Tinker also prices fine-tuning, which is the main reason to pick it over a plain API:
The data behind this chart
[
{
"label": "64K context",
"training_usd": "1.50"
},
{
"label": "256K context",
"training_usd": "3.00"
}
]Training costs $1.50 per million tokens at 64K and $3.00 at 256K.
Note the context sizes. Tinker lists 64K and 256K. It does not list the 1M window the card describes. A provider can serve a smaller window than the model supports, because every token of context needs KV cache memory on the provider's GPUs. vLLM's own recipe shows the same trade: it serves 262,144 tokens with a tensor-parallel size of 4, and says a 1M-token context needs 8. How context length works in Ollama covers the same idea for local models.
The hosted APIs named in the 12 November update are NVIDIA, Fireworks, Together, Baseten, Nebius, Modal and Cloudflare. This guide did not check their prices. Open each provider's own model page for its price and its maximum context before you build on it. To decide when a rented GPU beats paying per token, read GPU VPS vs API tokens: the break-even point.
Route 3: self-host the Apache 2.0 weights on a GPU cluster
The card lists two checkpoints, and the choice decides your hardware:
- NVFP4, the main checkpoint: at least 500 GB of aggregated VRAM. The card gives 2x NVIDIA B300 or 4x NVIDIA H200 as examples.
- MXFP8, the alternative: at least 800 GB of aggregated VRAM. The card gives 4x NVIDIA B300 or 8x NVIDIA H200 as examples.
The model was trained in NVFP4, which is its native precision. The card says: "We recommend the NVFP4 checkpoint for all use-cases." It describes the MXFP8 gain on evals as very marginal, about 1 to 2 percentage points. So MXFP8 needs about 300 GB more GPU memory for a small gain, and most clusters should use NVFP4.
Serve it with vLLM
The model card contains no launch commands. It links to a recipe for each inference engine it names: SGLang, vLLM, TokenSpeed and Unsloth. vLLM's own Inkling 2 Small recipe, as read on 13 November 2026, says support needs a vLLM nightly build, because no release includes it yet. Until one does, the recipe uses the vllm/vllm-openai:inkling-2-small container image. Its serve command for the NVFP4 checkpoint is:
vllm serve thinkingmachines/Inkling-2-Small \
--tensor-parallel-size 4 \
--max-model-len 262144 \
--max-num-seqs 256 \
--reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'Without a tensor-parallel flag, vLLM tries to load the model on a single GPU. The recipe lists its hardware as 8x B200 and says a 1M-token context needs a tensor-parallel size of 8. These flags come from a recipe page that changes often, so check it again on the day you deploy.
Serve it with SGLang
The model card has no SGLang command either, only a link to SGLang's cookbook recipe. That cookbook page says Inkling 2 Small support is not in an SGLang release yet, so it uses the lmsysorg/sglang:dev-inkling-2-small images. The cookbook sets parallelism by hardware. NVFP4 on B300, B200, GB300 or GB200 uses TP4 with EP4 (tensor parallel and expert parallel). H200 uses TP8 with EP8. It says EP should equal TP. The page builds the full launch command in an interactive panel, so take the exact flags from there.
If you launch the server with --host 0.0.0.0, it listens on every network interface, including the public one. Anyone who can reach its port can use your cluster. Block that port in the firewall, or bind the server to a private address that only your own services can reach.
The same network rule applies to vLLM. An inference server on a public port is an open API for anyone who scans for it.
Is there a GGUF or an Ollama version that runs locally?
A local GGUF exists, but it does not fit an ordinary VPS. Unsloth publishes multiple quantizations and running instructions, including -ngl 0 for CPU inference. Ollama lists only the cloud tag. llama.cpp merged Inkling 2 support into master through PR #32217 on 14 November 2026. llama.cpp release b12406 (15 November 2026) and later include Inkling 2 support from that PR.
Unsloth's GGUF builds need a llama.cpp build with Inkling 2 support; builds that only support the original Inkling will not load them. Its running guide still builds from open PR #32187 and does not confirm compatibility with the upstream release. The large-memory CPU exception starts with UD-IQ1_S, Unsloth's roughly 1-bit and lowest-quality quantization, at 171.7 GB of weights. Adding room for the KV cache, runtime and operating system puts the floor at roughly 180 GB or more of free RAM; that is arithmetic from the file size, not an Unsloth memory specification. A 594B MoE model on CPU is slow. To watch for a local version, check the Ollama tag list and the Quantizations list on the model's Hugging Face page; watch Unsloth's running guide and GGUF card for confirmation that its files load with an upstream llama.cpp release.
Where Inkling 2 Small is strong and where it trails
Every number in this section is self-reported by Thinking Machines. This guide has not reproduced them. The lab publishes two kinds of tables, and they answer different questions.
The first kind is in the launch post: short comparisons against four open-weight models (GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, Muse Spark Open and Kimi K3). There, Inkling 2 Small is highest on GDP.pdf, MMMU Pro, PerceptionBench and ScreenSpot Pro. It is not highest on every row. On CharXiv RQ with Python it scores 89.1% against Kimi K3's 91.3%. On Chartography with Python it scores 82.1% against DeepSeek's 83.0%. So the fair claim is narrow: on those named vision rows, it is the strongest open-weight model in the lab's own comparison.
The second kind is the full 13-model table, on the card and in the launch post. It adds closed models such as GPT-6 Luna, Gemini 3.8 Flash and Claude Haiku 5.5, and larger open models. The chart below sets Inkling 2 Small against one named model per row. The notes column names that model.
The data behind this chart
[
{
"label": "Finance Agent v2.1",
"inkling_2_small_pct": 67.5,
"comparator_pct": 65.8,
"notes": "Gemini 3.8 Flash, next best"
},
{
"label": "GDP.pdf",
"inkling_2_small_pct": 25.2,
"comparator_pct": 22.8,
"notes": "GPT-6 Luna, next best"
},
{
"label": "PerceptionBench",
"inkling_2_small_pct": 64.7,
"comparator_pct": 68.7,
"notes": "Gemini 3.8 Flash, leader"
},
{
"label": "MMMU Pro",
"inkling_2_small_pct": 81.6,
"comparator_pct": 85.6,
"notes": "Gemini 3.8 Flash, leader"
},
{
"label": "ScreenSpot Pro, original",
"inkling_2_small_pct": 83.8,
"comparator_pct": 85.4,
"notes": "Claude Haiku 5.5, leader"
},
{
"label": "ScreenSpot Pro, with Python",
"inkling_2_small_pct": 92.0,
"comparator_pct": 93.7,
"notes": "Gemini 3.8 Flash, leader"
},
{
"label": "DeepSWE v1.1",
"inkling_2_small_pct": 72.1,
"comparator_pct": 77.0,
"notes": "DeepSeek-V4.1-Flash-1017"
},
{
"label": "FrontierCode v2.0",
"inkling_2_small_pct": 25.4,
"comparator_pct": 34.9,
"notes": "Claude Haiku 5.5"
}
]In the full table, Inkling 2 Small is best on Finance Agent v2.1 at 67.5%, ahead of Gemini 3.8 Flash at 65.8%. It is also best on GDP.pdf at 25.2%, ahead of GPT-6 Luna at 22.8%. It is second on PerceptionBench and MMMU Pro, where Gemini 3.8 Flash leads with 68.7% and 85.6%. On ScreenSpot Pro, Claude Haiku 5.5 leads the original setting with 85.4%, and Gemini 3.8 Flash leads the with-Python setting with 93.7%. So the accurate summary is "the strongest open-weight model in the lab's comparison on vision". It is not "the best at vision".
Coding is good but not top. On DeepSWE v1.1 it scores 72.1%, against 77.0% for DeepSeek-V4.1-Flash-1017. MiniMax-M3.1-Flash (73.7%) and Gemini 3.8 Flash (73.8%) also score higher. On FrontierCode v2.0 it scores 25.4%, against 28.5% for DeepSeek-V4.1-Flash-1017 and 34.9% for Claude Haiku 5.5.
The Artificial Analysis Intelligence Index points the same way. The launch post's text gives a rounded score of 31. Its table gives the v5.0 figure:
The data behind this chart
[
{
"label": "Claude Haiku 5.5",
"aa_index_v5": 36.5
},
{
"label": "DeepSeek-V4.1-Flash-1017",
"aa_index_v5": 32.6
},
{
"label": "GLM 5.3",
"aa_index_v5": 31.7
},
{
"label": "Gemini 3.8 Flash",
"aa_index_v5": 30.8
},
{
"label": "Inkling 2 Small",
"aa_index_v5": 30.6
}
]Inkling 2 Small scores 30.6, the lowest of the 5 models in this chart.
The launch post names the work the model is built for. On images: OCR (optical character recognition, reading text from images), grounding, infographic interpretation and visual reasoning. On agents: calling tools over MCP (Model Context Protocol), plus agentic knowledge work and finance. The Finance Agent and GDP.pdf results support the finance and document side of that list.
Which route should you pick?
Pick by your hardest constraint.
- Prompts must stay on your own hardware, and you have a VPS. Inkling 2 Small does not fit an ordinary VPS. The large-memory CPU exception needs roughly 180 GB or more of free RAM for Unsloth's 171.7 GB, roughly 1-bit
UD-IQ1_Sweights plus KV cache, runtime and operating system overhead. Use the Inkling 2 capable llama.cpp build documented by Unsloth; its guide still builds from open PR #32187. llama.cpp release b12406 (15 November 2026) and later include Inkling 2 support from PR #32217, and Unsloth has not confirmed that its files load with the upstream release. CPU inference is slow, andUD-IQ1_Sis Unsloth's lowest-quality quantization. For ordinary small VPS plans, start with which AI models you can really self-host and Ollama vs llama.cpp for models that do fit. - You want to try it today from a small box. Use Ollama Cloud. It needs an account and an API key for direct API calls, and you pay per token. Your prompts go to Ollama's servers.
- You want to fine-tune it but have no cluster. Tinker prices training per token at 64K and 256K context.
- You have cluster capacity and need full control of the weights. Download the NVFP4 checkpoint from Hugging Face and serve it with vLLM or SGLang, using their current Inkling recipes.
FAQ
Can I run Inkling 2 Small on a VPS with Ollama?
Not locally on an ordinary VPS. As of 15 November 2026, Ollama lists only one tag, inkling-2-small:cloud, and that tag runs on Ollama's servers, not on your VPS. The VPS sends prompts and receives replies, so it needs no GPU. The large-memory CPU exception uses llama.cpp through Unsloth's documented build, not a local Ollama tag: roughly 180 GB or more of free RAM for the 171.7 GB UD-IQ1_S weights plus overhead. That roughly 1-bit file is Unsloth's lowest-quality quantization, and CPU inference is slow. llama.cpp release b12406 (15 November 2026) and later include Inkling 2 support from PR #32217; Unsloth's guide still builds from open PR #32187 and does not confirm that its files load with the upstream release.
How much GPU memory does Inkling 2 Small need?
The model card says the recommended NVFP4 checkpoint needs at least 500 GB of aggregated VRAM, for example 2x NVIDIA B300 or 4x NVIDIA H200. The MXFP8 checkpoint needs at least 800 GB, for example 4x B300 or 8x H200. The model has 594B total parameters. Only 17B are active per token, but memory must accommodate the full weights. Unsloth's GGUF quantizations support CPU inference in system RAM, so those official GPU memory floors do not apply to every deployment.
Is there a GGUF of Inkling 2 Small for llama.cpp?
Yes, but it does not fit an ordinary VPS. Unsloth publishes multiple GGUF quantizations and running instructions, including -ngl 0 for CPU inference. llama.cpp release b12406 (15 November 2026) and later include Inkling 2 support from PR #32217. Unsloth's guide still builds from open PR #32187 and does not confirm that its files load with the upstream release. Its smallest quantization, UD-IQ1_S, is roughly 1-bit, has 171.7 GB of weights and is the lowest-quality one it offers. The large-memory CPU exception needs roughly 180 GB or more of free RAM after adding room for KV cache, runtime and operating system overhead; that floor is arithmetic from the file size. A 594B MoE model on CPU is slow.
Does Inkling 2 Small really have a 1M-token context?
The model card says up to 1M tokens, and Ollama's library lists 1M. Providers can serve less. Tinker lists 64K and 256K. vLLM's own recipe serves 262,144 tokens with a tensor-parallel size of 4, and says 1M needs 8. Check each provider's maximum context before you rely on long inputs.