Run Inkling 2 Mini on a CPU VPS with llama.cpp
Inkling 2 Mini runs on a 32 GB CPU VPS with llama.cpp b12600 and a 17.6 GB GGUF. Which file to pick for your RAM, and why Ollama is Mac only for now.
Before you start: Mac readers and Inkling 2 Small
If you have a Mac with Apple silicon and 32 GB of memory or more, you do not need a VPS for this. Ollama runs Inkling 2 Mini on that Mac, and the Mac section near the end of this guide has the commands. If you came for the larger model, Inkling 2 Small, read whether you can run Inkling 2 Small yourself instead.
Can you run Inkling 2 Mini on a CPU VPS?
Yes. Inkling 2 Mini runs on a CPU-only VPS with 32 GB of RAM or more, through llama.cpp release b12600 or later and a 4-bit GGUF of about 17.6 GB. It does not run through Ollama's official tag on a Linux VPS yet, because Ollama v0.40.17 supports this model on Apple silicon only. A 16 GB VPS has 1-bit, 2-bit and 3-bit options, which need 9 to 15 GB in Unsloth's memory table, subject to context and system overhead, and those files cost real quality. Below 16 GB, the 1-bit and 2-bit files are options on machines with enough room for a short context and the operating system.
On a GPU server, vLLM and SGLang both serve the model from launch images built for it. Thinking Machines' own Tinker service lists Mini for training and sampling, with per-token prices given below.
Release, tag, file, price and support checks on this page were made on 21 and 22 November 2026. Several of them changed by the hour after launch. Check the release pages before you copy a version number.
What Inkling 2 Mini is
These facts come from the model card on Hugging Face, as of 21 November 2026.
- Released: 20 November 2026, by Thinking Machines Lab.
- Licence: Apache 2.0. The card also links the lab's Model Acceptable Use Policy.
- Input and output: text and image in, text out.
- Context window: up to 1M tokens.
- Parameters: 26.4B total, 3.4B active, and 5.4B effective active per token once the loop is counted.
- Architecture: a 24-layer, decoder-only, sparse MoE (mixture of experts). The middle 12 layers run three times with shared weights, which gives 48 layer applications per token. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers.
- Official checkpoint: one file of about 20 GB in mixed NVFP4 and MXFP8 precision. The Hugging Face file list shows a 19.5 GB weights file plus a 636 MB
mtp.safetensors. - Hardware the card names: one NVIDIA DGX Spark, Apple silicon with 32 GB or more, and other GPUs and platforms with 32 GB or more of VRAM recommended.
The names mislead, so here it is once: Inkling 2 Small is the bigger model, and Inkling 2 Mini is the 26.4B one. The card says Mini is less capable than Small across most of the lab's evaluations.
Other pages round the numbers. Ollama's library page says "26B parameters, 5.4B active", and its tag is named 26b-a5b. vLLM's materials say 3.5B active. This guide uses the card's figures. 3.4B is the active count for one pass, and 5.4B adds the extra passes through the looped layers, each of which adds about 1.0B.
The card does not list CPU-only servers. Its local-runtime list names llama.cpp "for quantized GGUF versions", and that is the route this guide takes. Running it on a CPU VPS is our reading of that support, not a setup the lab names. The card also warns: "Support for looped models in local runtimes is new, and speed varies by runtime and hardware." For how Mini fits among other open models, see which AI models you can self-host on a VPS.
Why a 26B model fits a 32 GB VPS, and what the loop costs
Fitting is about file size. The whole GGUF has to sit in RAM, even though each token uses only part of it. The 4-bit file from ggml-org is 17.6 GB, which leaves room on a 32 GB server for the operating system and the context cache.
Speed is a different question. A dense 27B model reads all of its weights for every token it generates. Inkling 2 Mini is a mixture of experts. In each layer a router picks 8 of 128 expert blocks for the token, and only those experts take part, plus the attention and other shared weights. The model therefore moves far less memory per token than a dense model with the same file size.
The loop pulls the other way. The middle 12 layers run three times, and each pass is real compute. The card says it plainly: "Because the middle layers run three times, compute per token is higher than the active-parameter count alone suggests." Each pass also keeps its own context state. vLLM's launch post describes its implementation this way: "The model has 24 layers but 48 KV caches and 48 conv-state caches." The KV cache holds the keys and values of earlier tokens, so it grows with context length. For the difference between this cache and a provider's prompt cache, see KV cache versus prompt cache.
So "3.4B active" understates both the compute and the context memory. The common rule of dividing memory bandwidth by weight size to predict tokens per second does not transfer cleanly to a looped MoE. Measure instead. The steps are below.
Route 1: run Inkling 2 Mini on a CPU VPS with llama.cpp
Pin llama.cpp at release b12600 or later
Pull request #32694, "Inkling 2 Mini", was merged into llama.cpp on 20 November 2026 at 22:01 UTC. Release b12600 was published at 23:30 UTC the same day, and GitHub's compare view reports it as identical to that merge commit. So b12600 and every later b-tag include Inkling 2 Mini support.
There is a trap here. GitHub marks the b-tags as pre-releases, so the repository's "latest release" points at v1.0.0, published on 13 November 2026. That build predates Inkling support. A reader who downloads "latest" gets a llama.cpp that cannot load this model. Pin a b-tag by name instead. Pinning llama.cpp releases on a server covers how to hold a version and upgrade on purpose. The tags are listed on the llama.cpp releases page.
Build from the tag. The CPU build needs only a compiler and CMake. llama.cpp's build guide says it builds without OpenSSL, but then has no SSL support. The -hf download used below fetches from Hugging Face over HTTPS, so install the OpenSSL headers too.
free -h
nproc
sudo apt update
sudo apt install -y build-essential cmake git libssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b12600
git describe --tags
cmake -B build
cmake --build build --config Release -j "$(nproc)"
ls build/bin/llama-serverfree -h should show a total of about 32 GB or more on the Mem: line. git describe --tags should print b12600. If it prints anything else, check the checkout before building. The last line should list build/bin/llama-server.
The follow-up fix, and why it does not block you
A second pull request, #32704, was merged on 21 November 2026 at 11:10 UTC. It fixes a bug where the second and third passes through the looped layers ran the NVFP4 experts without their per-expert scales. The PR reports that GGUFs carrying NVFP4 scales, such as Unsloth's, gave a perplexity of 130.43 where it should be 7.6531. It also says: "ggml quants are exactly as before so it does not affect them."
That last sentence is why b12600 is enough for this guide. The ggml-org and bartowski files use ordinary ggml quant types, which the bug does not touch. The newest release we checked on 21 November, b12606, was published before the merge, so no release tag we checked contained the fix at that time. Release b12628, checked on 22 November 2026, now contains the loop-slot fix. A related, generic fix for scale tensors, #32669, also merged on 21 November.
Pick the GGUF by your RAM
Three publishers have GGUF files, and they are not interchangeable. This is the part people get wrong.
- ggml-org/Inkling-2-Mini-GGUF is made for upstream llama.cpp. Its IQ4_NL file is imatrix-calibrated, with 4-bit experts in layers 0 to 19 and everything else at Q8_0.
- bartowski/thinkingmachines_Inkling-2-Mini-GGUF was quantized with llama.cpp release b12600, and its card says you need that release or newer. It has the widest range of sizes.
- unsloth/Inkling-2-Mini-GGUF stores its root files in Unsloth's own looped-layer layout. Unsloth's guide says those need Unsloth's own llama.cpp build, b12588-mix-c0c3de8. For mainline llama.cpp it points to a separate Mainline_Support folder. Do not load Unsloth's root files on an upstream llama.cpp release.
On upstream llama.cpp, use the ggml-org or bartowski files. Both repos have a file called IQ4_NL, and the two differ in size, so always name the repo with the file.
The data behind this chart
[
{
"label": "ggml-org IQ4_NL",
"file_gb": 17.6
},
{
"label": "ggml-org Q8_0",
"file_gb": 27.8
},
{
"label": "ggml-org BF16",
"file_gb": 52.2
},
{
"label": "bartowski IQ2_XXS",
"file_gb": 10.32
},
{
"label": "bartowski Q2_K",
"file_gb": 11.88
},
{
"label": "bartowski Q3_K_M",
"file_gb": 14.55
},
{
"label": "bartowski IQ4_NL",
"file_gb": 16.11
},
{
"label": "bartowski Q4_K_M",
"file_gb": 18.66
},
{
"label": "bartowski Q4_K_L",
"file_gb": 19.72
},
{
"label": "bartowski Q5_K_M",
"file_gb": 21.22
},
{
"label": "bartowski Q6_K",
"file_gb": "24.70"
},
{
"label": "bartowski Q8_0",
"file_gb": 27.82
},
{
"label": "bartowski IQ2_S",
"file_gb": 11.07
},
{
"label": "bartowski IQ2_M",
"file_gb": 11.94
}
]bartowski's sizes sit closer together than usual. The card explains why. 31.3% of the weights (the ffn_down_exps tensors, which are 1152 wide) cannot use the 256-wide K and IQ block types, so they get a 32-wide type whatever the file name asks for. A Q3 file therefore saves less against a Q4 file than you would expect.
The RAM you need is more than the file size. Unsloth publishes a memory table that counts RAM plus VRAM, or unified memory, and calls it a starting point: longer contexts, image inputs and other apps use more. The table is for Unsloth's files, but it shows the scale for each bit width.
The data behind this chart
[
{
"label": "1-bit",
"ram_low_gb": 9,
"ram_high_gb": 9
},
{
"label": "2-bit",
"ram_low_gb": 10,
"ram_high_gb": 11
},
{
"label": "3-bit",
"ram_low_gb": 13,
"ram_high_gb": 15
},
{
"label": "4-bit",
"ram_low_gb": 17,
"ram_high_gb": 18
},
{
"label": "8-bit",
"ram_low_gb": 24,
"ram_high_gb": 24
}
]Going small costs quality. Unsloth compared each of its files with its own UD-Q8_K_XL file across seven domains, and counted how often the smaller file picks the same top token. These figures are for Unsloth's files, which need Unsloth's build. Read them as the shape of the cost for the ggml-org and bartowski files, not as exact numbers for them.
The data behind this chart
[
{
"label": "UD-Q4_K_XL",
"same_top_token_pct": "98.0",
"file_gb": 17.63
},
{
"label": "UD-Q3_K_XL",
"same_top_token_pct": 96.3,
"file_gb": 13.63
},
{
"label": "UD-Q2_K_XL",
"same_top_token_pct": 91.6,
"file_gb": 9.14
},
{
"label": "UD-IQ2_XXS",
"same_top_token_pct": 90.6,
"file_gb": 8.49
},
{
"label": "UD-IQ1_S",
"same_top_token_pct": 89.4,
"file_gb": 7.76
}
]The 4-bit file keeps the same top token 98.0% of the time. The 2-bit and 1-bit files fall to about 90%, down to 89.4% for the smallest. ggml-org's card measures against BF16 on wikitext-2 instead, and reports 97.6% for its Q8_0 and 95.5% for its IQ4_NL. For what these quant names mean in general, see Q4 versus Q8 versus FP16 quantization.
So, by RAM:
- 32 GB: ggml-org IQ4_NL (17.6 GB) or bartowski Q4_K_M (18.66 GB). This guide is written for this setup.
- 48 GB or more: a Q8_0 file (27.8 GB from ggml-org) fits with room for context.
- 16 GB: bartowski's 2-bit files, including IQ2_XXS (10.32 GB), IQ2_S (11.07 GB), Q2_K (11.88 GB) and IQ2_M (11.94 GB), with a short context. Smaller 3-bit files are also options if context and system overhead fit. Unsloth lists 3-bit memory requirements at 13 to 15 GB for its files. Expect the quality drop shown above.
- Under 16 GB: Unsloth's 1-bit and 2-bit files start at 9 GB and 10 to 11 GB of memory respectively. They fit on some machines below 16 GB with a short context and enough room for the operating system; otherwise, pick a smaller model.
Run llama-server bound to 127.0.0.1
Start the server once by hand to check that the model loads. -hf downloads the file from Hugging Face on the first run.
./build/bin/llama-server \
-hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL \
--host 127.0.0.1 --port 8080 \
-c 8192 -n 4096 --no-mmprojEach flag has a reason:
--host 127.0.0.1keeps the server off the public internet. It is already the default, but writing it down stops a later edit from changing it by accident.--port 8080is set on purpose. The current llama-server README gives the default port as 9931, while some model cards still say 8080. Naming the port removes the doubt.-c 8192sets the context size. Its default is 0, which means "loaded from model", and for this model that is up to 1M tokens. Always set-cyourself.-n 4096caps the tokens per reply. The card says the model can get stuck in long reasoning loops, and it recommends limits on output tokens.--no-mmprojskips the vision projector. With-hf, llama-server downloads the projector automatically if the repo has one, and ggml-org's BF16 projector is another 642 MB. Leave the flag out if you want to send images.
In a second SSH session, check that it is ready:
curl -s http://127.0.0.1:8080/healthWhile the model loads, you get {"error": {"code": 503, "message": "Loading model", "type": "unavailable_error"}}. When it is ready, you get {"status": "ok" }. To use the built-in web UI from your laptop, open a tunnel with ssh -N -L 8080:127.0.0.1:8080 you@your-vps and browse to http://localhost:8080.
Once it works by hand, run it as a service. Running llama.cpp server on a VPS covers the systemd setup, so this guide does not repeat it. Keep the same flags in ExecStart.
The public bug reports so far are all GPU reports, as of 21 November 2026. Issue #32707 reports slow prefill and a crash with the ggml-org IQ4_NL file on an RTX 5090, and a commenter sees the same on Vulkan with AMD. Issue #32710 reports a perplexity of NaN on CUDA, and a commenter says the same prompt is correct on pure CPU. Issue #32732 is an abort on ROCm. Issue #32710 also includes pure CPU correctness and perplexity checks. Check the first answers you get before you trust it.
Measure the speed on your own VPS
Nobody has measured Inkling 2 Mini for this page, on any hardware, so this guide gives no tokens-per-second figure. The mechanism above explains why a guess would be unreliable. Measure it yourself with a repeatable tokens-per-second test for a local LLM. Record prompt processing and generation separately, because the loop affects both.
On a shared VPS, your result also depends on your neighbours. Check steal time while the test runs, as described in CPU steal time and noisy neighbours. A high st value means the host is giving your vCPU time to someone else, and your numbers will not repeat.
Start with -c 8192 and raise it in steps while you watch free -h. Stop well before the available column nears zero. Past that point the kernel either swaps or kills the process, and neither leaves you a usable server. llama-server picks its thread count automatically by default (-t defaults to -1). Try -t with a few values up to your nproc count, and keep the fastest.
Why ollama run inkling-2-mini is not the VPS route yet
Ollama's library has the model, as of 21 November 2026. It lists three tags: latest, 26b-a5b and 26b-a5b-nvfp4. All three share the digest d183b1321cb5, are 20 GB, have a 1M context window and take text and image input. The 26b-a5b-nvfp4 tag is labelled MLX, which is Apple's machine-learning framework. There is no :cloud tag. The listed capabilities are vision, tools and thinking.
The release notes for v0.40.17, the newest Ollama release on 21 November 2026, say: "inkling-2-mini is supported on Apple silicon only in this release. PC support will follow in an upcoming release." A Linux VPS is not Apple silicon, so this release does not support the model there. If a pull stops with requires MLX support, the requires MLX support error explains what it means.
Importing one of the GGUF files above into Ollama is not a confirmed route either. We found no Ollama page that says it works for this model. A Windows report that an Inkling 2 Mini GGUF fails to load with an unknown architecture was closed on 21 November, and we could not confirm a fix. Use llama.cpp directly for now. If you have only used llama.cpp through Ollama before, Ollama versus llama.cpp compares the two. Watch the Ollama releases page for the PC support the notes promise.
Route 2: a GPU server with vLLM or SGLang
The card recommends 32 GB or more of VRAM on GPUs other than DGX Spark and Apple silicon. Neither serving engine had the support in a release on 21 November 2026, so both use images built for this model.
vLLM's recipe says the model runs on NVIDIA Blackwell (B200, GB200, B300, GB300), Hopper (H100, H200) and DGX Spark. It says to use the vllm/vllm-openai:inkling-2-mini image until support is in a vLLM release. The recipe requires vLLM 0.35.0 or later, which was not released as of 21 November 2026. The newest release then was v0.34.0. The recipe's command:
docker run --gpus all --privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \
--tensor-parallel-size 1 --reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'-p 8000:8000 publishes port 8000 on the host. Read Docker's documentation on port publishing before you run this on a server with a public IP. Then either block port 8000 in your firewall or publish it as 127.0.0.1:8000:8000. The recipe caps the context at 262,144 tokens on H100 and DGX Spark, to fit in memory.
SGLang's cookbook lists B300/GB300, B200/GB200, H200, RTX PRO 6000, RTX 5090 and DGX Spark with the lmsysorg/sglang:inkling-2-mini image. AMD MI350X/MI355X use inkling-2-mini-mi35x, and TPU7x uses inkling-2-mini-tpu. Support was not in an SGLang release as of 21 November. The cookbook builds the launch command for your hardware with a picker, so copy it from the SGLang Inkling 2 Mini cookbook page. On a 32 GB RTX 5090, the commands checked on 21 November capped the context at 65,536 tokens, or 32,768 with DSpark speculative decoding. If the command you copy uses --host 0.0.0.0, it listens on every interface, so firewall the port or change the host to 127.0.0.1.
Whether a rented GPU is worth it depends on how many tokens you push through it. The GPU VPS versus API tokens break-even works through that sum.
Route 3: Tinker, the lab's own service
Tinker's pricing page, read on 21 November 2026, lists Inkling 2 Mini in its training and sampling table. The page header still says "as of October 19, 2026". Per 1M tokens, prefill costs $0.15, cached prefill $0.015 and sampling $0.50. Training costs $0.75 per 1M tokens with the 64K training context, and $1.50 with the 256K one. Tinker's docs say Mini samples up to 1M tokens on both training options.
Tinker's serverless inference, which is in beta, does not list Mini. Its docs say it "is available for Inkling and Inkling-Small only." The pricing page also lists Inkling 2 Small for serverless inference, but still does not list Mini. The model card adds that the Tinker playground gives inference-only access to the model. Confirm in Tinker's docs how you would sample Mini before you build on it as a plain pay-per-token endpoint.
On a Mac with 32 GB or more
This is the route the lab supports at launch. Ollama v0.40.17 runs the model on Apple silicon, and Ollama's macOS docs ask for macOS Sonoma (v14) or newer. The library page gives these commands:
ollama run inkling-2-mini
ollama launch claude --model inkling-2-mini
ollama launch opencode --model inkling-2-mini
ollama launch hermes --model inkling-2-mini
ollama launch openclaw --model inkling-2-miniThe ollama launch lines start a coding agent that uses the local model. Using Ollama with your coding agent covers that setup. The sandbox advice below applies on a Mac too.
Where Inkling 2 Mini is strong, and where it trails Qwen3.8-27B
The fair comparison for a VPS reader is Qwen3.8-27B, which the card's benchmark table includes. Its other columns are Gemma 4 31B, Muse Glimmer and Nemotron 3.5 Super VL. The card does not say who ran each row, though one group of rows sits under a "Third Party Index" heading. The chart below holds selected rows where both models have a score on the same kind of scale, including the rows Mini loses.
The data behind this chart
[
{
"label": "DeepSWE v1.1",
"inkling_2_mini": 48.2,
"qwen3_8_27b": 42.2
},
{
"label": "SWE-bench Pro v2 (Hard)",
"inkling_2_mini": 77.9,
"qwen3_8_27b": 73.8
},
{
"label": "SWE-bench Verified (bash only)",
"inkling_2_mini": 79.2,
"qwen3_8_27b": 77.8
},
{
"label": "MCP Atlas",
"inkling_2_mini": 82.7,
"qwen3_8_27b": 77.3
},
{
"label": "DeepSearchQA",
"inkling_2_mini": 85.3,
"qwen3_8_27b": 79.7
},
{
"label": "ScreenSpot Pro",
"inkling_2_mini": 76.5,
"qwen3_8_27b": 57.6
},
{
"label": "OmniDocBench v1.6",
"inkling_2_mini": 92.8,
"qwen3_8_27b": 91.5
},
{
"label": "LongBench v2",
"inkling_2_mini": 66.9,
"qwen3_8_27b": 52.3
},
{
"label": "AA Index v5.0",
"inkling_2_mini": 20.1,
"qwen3_8_27b": 20.1
},
{
"label": "Terminal Bench v4.0",
"inkling_2_mini": 5.6,
"qwen3_8_27b": 5.6
},
{
"label": "MMMU Pro",
"inkling_2_mini": 74.9,
"qwen3_8_27b": 76.3
},
{
"label": "Terminal Bench v2.1",
"inkling_2_mini": 71.6,
"qwen3_8_27b": "73.0"
},
{
"label": "AutomationBench-AA",
"inkling_2_mini": 41.8,
"qwen3_8_27b": 48.2
},
{
"label": "EnterpriseOps-Gym",
"inkling_2_mini": "42.0",
"qwen3_8_27b": 44.2
},
{
"label": "AA-Omniscience non-hallucination rate",
"inkling_2_mini": 63.4,
"qwen3_8_27b": 71.4
}
]Mini leads on coding: DeepSWE v1.1 48.2 against 42.2, and SWE-bench Pro v2 (Hard) 77.9 against 73.8. It leads on tool use and search, with MCP Atlas at 82.7 against 77.3. The widest gaps are on screens and long context: ScreenSpot Pro 76.5 against 57.6, and LongBench v2 66.9 against 52.3.
The two tie on the Artificial Analysis Intelligence Index v5.0 (20.1 each) and on Terminal Bench v4.0. Qwen3.8-27B leads on MMMU Pro, Terminal Bench v2.1, AutomationBench-AA and EnterpriseOps-Gym. It also leads on GDPVal-AA, 1427 against 1319. That score uses a different scale, so it is left out of the chart.
The row to read twice is the last one. Qwen3.8-27B's non-hallucination rate on AA-Omniscience is 71.4% against Mini's 63.4%. That matches the card's own warning that Mini states wrong facts with confidence more often than larger models. If your work is factual lookup, test both. Running Qwen3.8-27B on a VPS covers the other model.
What the card says Inkling 2 Mini does badly
Section 7 of the card lists these limitations. They belong in any decision to deploy it.
- It is less capable than Inkling 2 Small across most evaluations, with the largest gaps on factual recall and knowledge-heavy questions.
- It is more likely than larger models to state incorrect information with confidence.
- In the lab's testing it sometimes got stuck in long reasoning loops. The lab recommends limits on output tokens and agent turns, which is why the server command above sets
-n. - With tools, it may not follow detailed instructions or project conventions. It "may take destructive actions, such as deleting or overwriting files, without confirmation", and it may not report the outcome of its work accurately.
- Output quality can be lower outside English. In Chinese, the lab saw clipped phrasing and mixed languages.
- Setting reasoning effort to its minimum does not fully turn reasoning off.
- Because the middle layers run three times, compute per token is higher than the active-parameter count suggests.
Run a coding agent on it only inside a sandbox
Several rows where Mini leads are coding and tool-use rows, so you may want to point a coding agent at it. Do that inside a sandbox. The card itself recommends running agentic workloads in a sandbox and requiring confirmation before destructive actions. The reason is in its limitations: the model may delete or overwrite files without asking, and then report a result it did not achieve. A throwaway VM holds only what the agent needs, so a bad command costs you a rebuild and nothing more. Running coding agents in a disposable VM shows the setup. Cap the agent's turns as well as its output tokens.
FAQ
Is Inkling 2 Mini the same model as Inkling 2 Small?
No. Despite the names, Inkling 2 Small is the larger model and Inkling 2 Mini is the smaller one, with 26.4B total parameters and 3.4B active. The Mini card says it is less capable than Small across most of the lab's evaluations, with the largest gaps on factual recall. Tinker's pricing page lists the two as separate models.
Can I run Inkling 2 Mini on a 16 GB VPS?
Yes, with a low-bit file and a short context. Unsloth's memory table puts 1-bit files at 9 GB, 2-bit files at 10 to 11 GB and 3-bit files at 13 to 15 GB, before longer contexts, images and system overhead. On upstream llama.cpp, bartowski's IQ2_XXS (10.32 GB), IQ2_S (11.07 GB), Q2_K (11.88 GB) and IQ2_M (11.94 GB) are candidates; smaller 3-bit files are also options if the remaining memory covers context and system overhead. In Unsloth's own comparison, its 2-bit files pick the same top token as its 8-bit file only about 91% of the time, against 98.0% for 4-bit, so expect weaker answers. Some machines below 16 GB also fit Unsloth's 1-bit or 2-bit files with a short context and enough room for the operating system.
Does ollama run inkling-2-mini work on a Linux VPS?
Not as of 21 November 2026. The release notes for Ollama v0.40.17 say the model "is supported on Apple silicon only in this release", and that PC support will follow in an upcoming release. Importing a GGUF into Ollama is not confirmed to work for this model either. On Linux, use llama.cpp release b12600 or later, and watch Ollama's release notes for PC support.
Which GGUF should I download for upstream llama.cpp?
Use the ggml-org or bartowski files. On a 32 GB server, ggml-org's IQ4_NL (17.6 GB) or bartowski's Q4_K_M (18.66 GB) is the right size. Do not use the files in the root of Unsloth's repository on an upstream release, because they use Unsloth's looped-layer layout and need Unsloth's own llama.cpp build. ggml-org and bartowski each have a file named IQ4_NL with a different size, so always name the repo with the file.
How fast is Inkling 2 Mini on a CPU VPS?
This guide gives no measured CPU VPS figure, so measure it on your own server. Its mixture-of-experts design uses only 8 of 128 experts per layer for each token, which reduces the memory moved per token. Its middle 12 layers run three times, which adds compute and context memory. Run the same prompt several times at a fixed -c value. Record prompt processing and generation speed separately. Watch the st value in top while it runs, because a busy shared host lowers your numbers.