How to Run Nemotron 3.5 Lightning on VPS

Wetín Nemotron 3.5 Lightning dey for

Nemotron 3.5 Lightning na NVIDIA open 30B mixture-of-experts model wey dem release for August 2026. Dem build am for agents wey dey run for hours, no be just one chat window. MoE (mixture of experts) mean say dem split the weights into plenty expert sub-networks, and each token pass through only small number of dem. NVIDIA model card talk say e get 30 billion total parameters, but na 3 billion active for each token. You go pay memory cost for the big number. You go get speed benefit from the small number.

Na this trade-off make this model worth considering for server wey you rent. Agent wey dey do real work fit send thousands of short requests across one day, so throughput per dollar go decide whether e fit run for your own machine. Model wey dey take 40 seconds for each reply fit work as assistant, but e no good as agent. One task fit make twenty calls, and you go wait for every one.

NVIDIA describe the architecture as hybrid: Mamba-2 and MoE layers dey alternate, with select attention layers. The model card give maximum context length up to 1M tokens and OpenMDW-1.1 license, and e mark am ready for commercial use. The main languages na English and code. Spanish, French, German, Italian, and Japanese dey listed too.

Artificial Analysis publish launch measurements for August 2026. The result show nearly 670 output tokens per second, for pre-release DeepInfra endpoint wey dey serve the NVFP4 weights. Na hosted GPU endpoint be that. Understand am as wetin the architecture fit allow, no be as wetin your VPS go necessarily achieve.

Which Ollama tag fit which VPS

The Ollama library dey publish different builds of the same weights. Wetin dey change between dem na quantisation, wey mean how many bits dem dey use store each weight. This one dey change download size plenty.

[
  {
    "label": "30b-a3b-q4_K_M",
    "size_gb": 25
  },
  {
    "label": "30b-a3b-q8_0",
    "size_gb": 35
  },
  {
    "label": "30b-a3b-bf16",
    "size_gb": 66
  },
  {
    "label": "30b-a3b-mlx",
    "size_gb": 23
  }
]

The tags wey dem name latest, 30b and 30b-a3b all resolve to the same digest as 30b-a3b-q4_K_M. So, the default download na the 25 GB four-bit build with the full 1M context. Q8_0 na 35 GB and bf16 na 66 GB. Both still dey use 1M context. The MLX builds wey be 23 GB na for Apple silicon and dem cap at 256K context. So, dem no be the correct choice for Linux VPS.

Those sizes na download sizes, no be memory requirement. NVIDIA no publish minimum VRAM (video RAM) figure for the Ollama builds. So, treat the download size as the minimum floor, nothing more. The weights must dey resident somewhere. If the GPU card fit hold dem, dem go dey GPU memory. Otherwise, dem go dey system RAM. The KV cache (key/value cache, wey be the model memory for each token of the conversation) dey add on top. The real number for your hardware go come from command, no be arithmetic, and e dey below. If you never choose quantisation level, wetin Q4, Q8 and FP16 each go cost you explain wetin you go give up for each step.

Pull the exact tag, never latest

latest na moving pointer. When the library republishes am, your agent behaviour go change for the next pull, and your notes no go explain why. Name the tag.

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
ollama pull nemotron-3.5-lightning:30b-a3b-q4_K_M

The install script go set up a systemd service wey dey run as the ollama user and keep models for /usr/share/ollama/.ollama/models. For most VPS images, that path dey for the root filesystem. So check say space dey before you request 25 GB.

df -h /usr/share/ollama

If pull stop halfway and e report no space left on device, na exactly that e mean. The partial blobs go remain for disk until you delete dem. Then confirm wetin land:

ollama show nemotron-3.5-lightning:30b-a3b-q4_K_M

ollama show go print the architecture, parameter count, context length, and quantisation wey the file actually carry. If any of dem no match the library page, you pull different tag from the one wey you intend.

Run am, then check where e actually run

sudo systemctl enable --now ollama
ollama run nemotron-3.5-lightning:30b-a3b-q4_K_M "Reply with one word: ready"

While the model still dey loaded, open second shell:

ollama ps

Na this command go answer the memory question for your machine. ollama ps dey show the model wey load, the size wey e dey occupy for memory, and one PROCESSOR column. 100% GPU mean say everything dey for VRAM. 100% CPU mean say nothing dey for VRAM, and processor dey compute every token from system RAM. Split like 65%/35% CPU/GPU mean say all the layers no fit, and the CPU share dey determine your speed. No estimate the requirement. Load am and read this line.

If e no fit load at all, Ollama go refuse cleanly instead of crashing:

Error: model requires more system memory (28.4 GiB) than is available (15.6 GiB)

CPU-only VPS dey fast enough?

General purpose VPS no get GPU, so CPU go do all the work and read every weight wey e need from system RAM. MoE help for here, because na about 3 billion out of the 30 billion parameters dey touched for each token. So the arithmetic per token much smaller than wetin dense 30B model need. Memory no get any help at all. All 30 billion parameters must remain resident, because router fit choose any expert for any token.

So CPU-only inference for this model dey limited by memory bandwidth, no be core count. Adding vCPUs to plan wey already get reasonable number of dem no go change much. Wetin you need na enough RAM to hold the weights plus your KV cache, and the fastest memory wey the plan provide.

Measure am before you commit agent to am, using the method for measuring tokens per second for local LLM:

ollama run --verbose nemotron-3.5-lightning:30b-a3b-q4_K_M "Write a 200 word summary of TCP slow start."

The eval rate line wey print for the end na your generation speed for tokens per second. Na this one number decide the question, because agent wall-clock time mostly depend on am.

[
  {
    "label": "Nemotron 3.5 Lightning",
    "sec_per_task": 30
  },
  {
    "label": "gpt-oss-120b",
    "sec_per_task": 204
  },
  {
    "label": "Qwen3.6 35B",
    "sec_per_task": 210
  }
]

Na published figures from third parties, converted from the per-task minutes wey Artificial Analysis report when dem launch am. Dem measure am for hosted GPU endpoints, no be VPS. Nemotron 3.5 Lightning average about 30 seconds per task. gpt-oss-120b take roughly 204, while Qwen3.6 35B take roughly 210. Use dem to understand the size of the gap, no be promise about your hardware.

The honest advice depend on who dey wait. If person dey wait for the agent, or the agent dey make long chains of calls one after another, rent GPU capacity. If e dey run overnight by schedule and nobody dey watch am, large-RAM CPU plan fit work well. Either way, setup na the same, and running Ollama on a VPS explain plan sizing and how GPU instance compare with paying API provider per token. The break-even na utilisation question: GPU instance dey bill every hour wey e exist, while API tokens dey bill only when dem use am. So agent wey busy most of the day favour the box wey you own, while agent wey fire two times for one hour usually no favour am.

The 1M context window no be free

1M tokens na the model maximum, and Ollama no dey give you that by default. Ollama dey serve much smaller default window and e dey drop the oldest tokens once conversation pass that limit. Nothing dey log when this happen, so for agent e go look like say model don forget the beginning of its own task.

Set the window intentionally. For the whole server, edit the service:

sudo systemctl edit ollama

Add this, then run sudo systemctl restart ollama:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

For each request, send num_ctx inside the options object instead:

curl http://localhost:11434/api/chat -d '{
  "model": "nemotron-3.5-lightning:30b-a3b-q4_K_M",
  "messages": [{"role": "user", "content": "Say ready"}],
  "options": {"num_ctx": 32768},
  "stream": false
}'

Every increase dey cost memory, because KV cache dey grow based on the number of tokens wey you allow. Increase the value, restart, then run ollama ps again and monitor the reported size as e dey increase. If the PROCESSOR column change from 100% GPU to a split after that change, KV cache don push model layers comot from VRAM and your speed go drop seriously. How to choose num_ctx for Ollama explain this trade-off well. No set 1000000 just because model card allow am, because allocation dey happen upfront and the load go simply fail.

Connect am to agent wey dey always on

Ollama launch post for this model document one shortcut wey go start supported agent and already point am to the model:

ollama launch claude --model nemotron-3.5-lightning

The post document claude, opencode, openclaw and hermes for that position. The subcommand need current Ollama, so check ollama --version first. If e no dey, point the agent to the API by yourself. Ollama expose OpenAI-compatible endpoint, wey most agent harnesses accept:

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=ollama

Ollama no use the key, but most clients no go start unless you set one. The harness side of this dey covered for point coding agent to Ollama and build your own OpenClaw agent.

Two server settings matter when the agent dey run unattended. OLLAMA_KEEP_ALIVE control how long model go remain for memory after the last request. The default unload am after five minutes, so the next call go pay the full load time again. For a 25 GB file without GPU, that pause fit long enough to make timeout happen. Set OLLAMA_KEEP_ALIVE=-1 to keep am resident. OLLAMA_HOST=0.0.0.0:11434 make the API reachable from other machines, and e no get authentication at all. So open am only behind firewall rule or private network.

Failure modes, with the strings you go see

The pull fails immediately. Error: pull model manifest: file does not exist mean say that tag no dey exist. Tag names na exact strings, so copy one from the library page instead of guessing quantisation suffix.

The model will not load. Error: model requires more system memory (28.4 GiB) than is available (15.6 GiB) mean say the tag too big for this plan as e dey configured. Move go smaller quantisation, or reduce OLLAMA_CONTEXT_LENGTH, because KV cache dey counted inside that requirement.

Nothing answers on port 11434. curl: (7) Failed to connect to localhost port 11434 mean say the service no dey run, or e no dey listen where you expect. Read systemctl status ollama and journalctl -u ollama -n 50. If you also start ollama serve by hand, the second copy go exit with Error: listen tcp 127.0.0.1:11434: bind: address already in use.

It answers, very slowly. Check ollama ps before you change anything. Any CPU share for PROCESSOR column on GPU machine mean say part of the model spill out of VRAM, so reduce the context or use the smaller quantisation. For machine wey no get GPU, slow na the expected result and no setting fit repair am.

The agent forgets its instructions partway through a task. The conversation don pass the context window, and the oldest tokens discard silently. Increase OLLAMA_CONTEXT_LENGTH, confirm with ollama ps say the model still fit, and if e no fit again, the fix na bigger machine instead of smaller window.

Wia dis model stand compared with di alternatives

30B MoE big well well to host for small work. If dense 8B model already fit handle your task, e go cost much less to run and load within seconds. Qwen 3 for 8B and 27B on a VPS na direct comparison for that decision. If you want wider check of wetin one plan fit actually hold, start from which AI models you fit self-host. If you plan serve several agents at once instead of one, read Ollama compared with vLLM first. Ollama no dey batch concurrent requests the way production inference server dey do am. Na for this point single-user setup stop to scale.

FAQ

Nemotron 3.5 Lightning tag wey I suppose pull for Linux VPS na which one?

Use nemotron-3.5-lightning:30b-a3b-q4_K_M. E get 25 GB, e carry the full 1M maximum context, and na the same digest wey latest, 30b and 30b-a3b tags point to as of August 2026. Name am explicitly instead of pulling latest, so if dem republish that pointer later, e no go change your agent behaviour without you noticing. mlx tags na Apple silicon builds, and dem no go help you for Linux.

How much RAM Nemotron 3.5 Lightning need?

NVIDIA no publish minimum memory figure for the Ollama builds, so measure am instead of estimating. Pull the tag, run the model once, then read ollama ps while e dey loaded. E go show the size wey e actually occupy and whether e land for GPU or CPU. The download size, 25 GB for the default tag, na only the floor, because KV cache dey add on top and e dey grow with the context window wey you set. If the plan too small, Ollama go refuse with model requires more system memory and name both numbers.

I fit run Nemotron 3.5 Lightning for VPS wey no get GPU?

Yes, if the plan get enough RAM to hold the weights. The MoE design help because na about 3 out of the 30 billion parameters dem dey compute for each token. Speed na the main problem. Without GPU, memory bandwidth dey limit the model, so adding vCPUs hardly change the result. Run ollama run --verbose with a fixed prompt, read the eval rate line, then compare that number with your agent deadline. For batch job wey go run overnight, e often dey okay. For anything wey person dey wait for, e usually no dey okay.

Why Ollama no dey give me the full 1M context window?

1M na the model maximum, e no be Ollama default. Ollama dey use much smaller window and discard the oldest tokens once conversation pass am. E no print error, so e fit look like say the agent don forget its own instructions. Set OLLAMA_CONTEXT_LENGTH for the systemd service, or pass num_ctx for each request. Increase am step by step and check ollama ps again every time, because KV cache memory dey scale with the window and fit push model layers comot from GPU.

Nemotron 3.5 Lightning free to use commercially?

NVIDIA model card put the model under OpenMDW-1.1 license and mark am ready for commercial use. This cover the weights wey you download and run by yourself. E no talk about the other software for your stack, so check the licences of the agent harness and any tools wey you connect to am separately. Also read the current model card before you rely on this for anything contractual.