Which AI model fit run for your VPS RAM?

Wetin dey decide which AI models you fit self-host

Na one number dey decide which AI models you fit self-host: the RAM wey dey for the box. The model family and framework no matter pass whether the weights fit enter memory and still leave some space. This post na the arithmetic wey you go use calculate am. Installing runtime na separate work, and this guide to running Ollama on a VPS cover am.

Two costs dey decide the answer. The weights na fixed cost, wey parameter count and quantisation set. The context window na running cost. Na this one people dey forget until model wey load yesterday refuse load today.

Sizing arithmetic: bits per parameter

Model file dey almost entirely consist of weights. Each weight dey store with some number of bits. Quantisation mean say you store dem with fewer bits than the precision wey dem use train the model, so you lose small accuracy but save plenty memory. The size follow directly from this:

weights in GB = (parameters in billions x bits per weight) / 8

Dem dey release models with 16 bits, wey be 2 GB for every billion parameters. Na why almost nobody dey run release precision for VPS. Na these quantisations you go actually meet, with their real average bits per weight:

• Q8_0 dey store about 8.5 bits per weight, so e dey roughly 1.1 GB for every billion parameters.
• Q6_K dey store about 6.6 bits, so e dey roughly 0.83 GB for every billion.
• Q5_K_M dey store about 5.7 bits, so e dey roughly 0.71 GB for every billion.
• Q4_K_M dey store about 4.8 bits, so e dey roughly 0.6 GB for every billion.

Use 0.6 GB for every billion parameters as your working number. Q4_K_M na sensible default for box wey memory dey limit am: quality loss compared with 8 bits small for most tasks, and the file nearly half the size. Below 4 bits, the loss dey increase quickly, so 70B wey dem squeeze to 2 bits usually dey answer worse than 32B for 4 bits from the same generation. When memory tight, reduce the size class before you go below 4 bits.

[
  {
    "label": "3B",
    "weights_gb": 1.8,
    "kv_8k_gb": 0.9,
    "kv_128k_gb": 14
  },
  {
    "label": "8B",
    "weights_gb": 4.8,
    "kv_8k_gb": 1,
    "kv_128k_gb": 16
  },
  {
    "label": "14B",
    "weights_gb": 8.4,
    "kv_8k_gb": 1.5,
    "kv_128k_gb": 24
  },
  {
    "label": "32B",
    "weights_gb": 19.2,
    "kv_8k_gb": 2,
    "kv_128k_gb": 32
  },
  {
    "label": "70B",
    "weights_gb": 42,
    "kv_8k_gb": 2.5,
    "kv_128k_gb": 40
  }
]

The weight column above na that 0.6 GB-per-billion rule wey dem apply. Real GGUF files dey fall within few percent of am, because dem keep embedding and output layers for higher precision than the remaining layers. 3B model for 4 bits na about 1.8 GB. 8B na 4.8 GB. 32B na 19.2 GB, and 70B na 42 GB.

Why context length dey cost more RAM pass weights

The KV cache (key value cache, the attention state wey model dey keep for every token wey currently dey inside conversation) na the second cost. E dey allocate am when model load, size am based on context length wey you ask for, and e dey grow straight with that length.

bytes per token = 2 x layers x kv_heads x head_dim x bytes per element

For Ollama default context, that 8B model dey use half gigabyte for cache. For 8192 tokens e dey use 1 GB. For the 128k context wey the model card advertise, e dey use 16 GB, wey pass three times the weights. The 70B na opposite case: e cache for 128k na 40 GB, less than the weights wey belong to am, because grouped query attention dey stop the per token cost from growing anywhere near as fast as the parameter count.

Ollama default context length na 4096 tokens for server wey na CPU only. When GPU dey available, e dey choose default from VRAM instead: 32k between 24 and 48 GiB, and 256k for 48 GiB and above. Raise am with OLLAMA_CONTEXT_LENGTH variable for the server, then check wetin running model actually get for CONTEXT column of ollama ps. The memory calculation behind that setting dey explained for the post about num_ctx and context length.

You get two ways to reduce the cache again. Ask for the context wey you need instead of the context wey model card advertise, because most chat and coding work fit inside 8k to 32k. Or quantise the cache itself to 8 bits, wey go halve am, but e fit reduce recall for long context.

Resident model dey hold RAM until something unload am

Ollama dey keep model for memory for 5 minutes after the last request, then e unload am. That default fit laptop, but e no good for server, because the first request after every idle gap go pay the load time again.

ollama ps
ollama stop qwen3:4b

ollama ps dey show wetin dey resident, with SIZE column wey dey show how much memory e dey hold and UNTIL column wey dey show when e go expire. To pin model permanently, set OLLAMA_KEEP_ALIVE=-1 for the service. Value of 0 go unload am as soon as each response finish.

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Send one prompt, then run ollama ps again 10 minutes later. The model still dey listed, and na exactly the point be that: e dey hold that RAM whether anybody dey use am or not. Pinned model no be spare capacity. For 16 GB VPS, 8B for 8k context dey hold roughly 6 GB for as long as the service dey run, so size the box based on the model plus your application, no be the model alone. Pinning model for memory explain the trade-off against cold start latency.

Wetin fit run for a 4 GB VPS

Keep about 1 GB for the operating system and the model server. Dis one leave roughly 3 GB. That one fit carry 1B to 4B model for 4 bits, with the default 4096 token context. As of August 2026, that class include Llama 3.2 for 3B, Qwen 3 for 1.7B and 4B, plus the small Gemma and Phi releases. Take dem as size examples, no be recommendations. The names dey change every few months, but the arithmetic no dey change.

Expect roughly 6 to 14 tokens per second. Models wey small like dis dey handle narrow work well: classification, tag extraction, short summaries, and rewriting one paragraph to match house style. Dem no too strong for multi step reasoning or code wey span several files, and no amount of prompting go fix that.

The main failure for dis tier na swap. If the model no fit for memory, Linux no go refuse to load am. Instead, e go page memory out to disk. Since generating one token reads every weight once, generation go slow down reach seconds per token. Monitor free -h and the si and so columns of vmstat 1 while the model dey answer. If swap in and swap out show non-zero values during generation, the model too big for the plan.

Wetin fit run for 8 to 16 GB VPS

Na for here self-hosted model dey become generally useful. For 8 GB, you fit run 7B or 8B for 4 bits, with about 4.8 GB weights, plus 8k context. For 16 GB, you fit run 13B or 14B for 4 bits, with about 8.4 GB, or keep 8B for 8 bits if you prefer use the memory for more precision instead of more parameters.

Speed na the main problem. 8B for CPU dey generate about 3 to 7 tokens per second, while 14B dey generate about 1.5 to 3.5. Person dey read around 5 to 10 tokens per second, so 8B for CPU VPS fit feel like say you dey watch slow typist. E good for background job, but e fit tire person for interactive chat. Measured runs of Qwen 3 at 8B and larger on a VPS show how e dey work for real life.

Wetín fit run for a 32 to 64 GB VPS

A 32B for 4 bits na about 19.2 GB, so e fit enter 32 GB plan with short context and e go get enough space for 48 GB or 64 GB. A 70B for 4 bits na about 42 GB, so e need 64 GB before you add any cache at all.

Then check the speed as e really be. A 32B for CPU dey run around 0.6 to 1.5 tokens per second, and a 70B dey run 0.2 to 0.5. One 500 token answer from that 70B go take about twenty minutes. These tools dey work well for batch jobs. Give dem queue of documents make dem process overnight, and speed no too matter. Put dem behind chat window, and speed go matter well well.

Mixture of experts routing dey change this calculation, and na the one architecture detail wey worth learning. An MoE model dey send each token through only small part of its weights. Model wey get 30B total parameters and 3B active per token need the memory of 30B, but e dey generate almost with the speed of dense 3B, because each token dey read only the active experts. For 32 GB box, MoE with this kind setup dey far more usable than dense 30B. The rule wey you suppose remember be say: total parameters set the memory, while active parameters set the speed.

CPU inference dey fast reach where, honestly?

To generate one token, system need read every active weight from memory once. Nothing fit avoid this, so generation speed for CPU dey depend on memory bandwidth, no be number of cores. The maximum na usable memory bandwidth divide by weight size for bytes. Small shared VPS normally fit deliver 10 to 25 GB per second across e vCPUs, so 4.8 GB model fit reach about 2 to 5 tokens per second.

[
  {
    "label": "3B",
    "tokens_per_second_low": 6,
    "tokens_per_second_high": 14
  },
  {
    "label": "8B",
    "tokens_per_second_low": 3,
    "tokens_per_second_high": 7
  },
  {
    "label": "14B",
    "tokens_per_second_low": 1.5,
    "tokens_per_second_high": 3.5
  },
  {
    "label": "32B",
    "tokens_per_second_low": 0.6,
    "tokens_per_second_high": 1.5
  },
  {
    "label": "70B",
    "tokens_per_second_low": 0.2,
    "tokens_per_second_high": 0.5
  }
]

Na ranges wey people commonly report for ordinary VPS hardware; e no be benchmark for one particular machine. Your own result depend on memory generation, number of channels for the host, and how many neighbours dey compete for the same resource. Measure your own result with any model tag wey you already get:

ollama run qwen3:4b --verbose "Write three sentences about disk latency."

The summary wey e print after answer finish get one line wey read eval rate: ... tokens/s. Na your generation speed be that. Ignore the first run for one session, because load duration for the same summary include the time wey e take read weights from disk. How to measure tokens per second properly explain how to get number wey worth comparing.

Two results fit surprise people here. Adding vCPUs stop to help quickly, because after roughly 8 cores, the extra cores dey wait for memory instead of doing arithmetic. And for shared plan, the same command fit return different numbers from one hour to another. Na CPU steal time from noisy neighbour cause this, no be wrong configuration wey you set.

Reading your prompt na different work from generating the answer. Prompt processing dey limited by compute, so e dey scale with cores, and na here GPU get the biggest advantage. CPU fit take minutes to read long document, while GPU fit take seconds. Na the first wall you go hit when pointing coding agent to model wey you host, because every turn resend file context and tool definitions before even one answer token come back.

Wetin dey change when you add GPU

The arithmetic no change; na the pool wey e apply to dey change. VRAM na hard limit, so calculate wetin fit enter before you rent am:

• 8 GB of VRAM fit hold 7B or 8B for 4 bits with short context.
• 16 GB fit hold 14B for 4 bits with real context, or 8B for 8 bits.
• 24 GB fit hold 32B for 4 bits when you keep the context short.
• 48 GB and above fit hold 70B for 4 bits with space for cache and concurrency.

When model no fit, Ollama go split am: some layers for GPU, the rest for CPU. ollama ps dey report the split for im PROCESSOR column, like 78%/22% CPU/GPU. Take am as warning, no be feature. The CPU half dey set the speed, because every token still dey wait for those layers. So, model wey get one-quarter of im layers for CPU go run closer to CPU speed than GPU speed. If you see split wey you no plan, reduce context length first. Na usually cache push am pass the limit.

Concurrency na another reason to size up. Requests wey dey happen at the same time share the weights, but every active request need im own KV cache. So, ten concurrent users of 8B for 8k context need ten times 1 GB of cache on top of the weights. How to serve concurrent users from one self-hosted model explain where that ceiling dey.

Whether GPU worth renting still na arithmetic question. E depend on how many tokens you really generate every month. The break-even point between GPU VPS and API tokens get those numbers.

Wetin you no fit self-host

Two different barriers dey here, and e go help if you know which one you dey face.

The first one na closed weights. Dem no distribute frontier commercial models, so no file dey to download, and any amount of RAM no go change that. You fit self-host everything around dem: the interface, the retrieval layer, the agent loop, and the logs. But the model itself go remain remote API. Whether you fit self-host Claude explain everything about this.

The second one na open weights wey too big. The biggest open releases na mixture of experts designs wey get hundreds of billions of total parameters. The same rule apply to dem: 400B total parameter model for 4 bits need about 240 GB for weights alone, before any cache. Na specialist hardware be this, and to rent am every month cost far pass wetin most people dey spend for API tokens in one year. Wetin e take to self-host Kimi class model explain the real requirement.

The honest line between the two be this: self-host when the load dey steady and the data no suppose leave your server. Buy tokens when the load dey bursty, or when na frontier answer quality be the thing you really need.

Check wetin you get before you choose

free -h
nproc
lscpu | grep 'Model name'

Plan based on the available column for free -h, no be the total column, because total includes memory wey the system dey already use. Remove about 1 GB for the operating system and the model server. Divide wetin remain by 0.6 to get the biggest parameter count in billions wey you fit hold for 4 bits. Then remove the KV cache for the context wey you actually want. Wetin remain na your answer, and unlike list of model names, e no go become outdated.

FAQ

How much RAM I need to run an 8B model?

About 4.8 GB for the weights with 4 bit quantisation, plus KV cache for your context length, plus roughly 1 GB for the operating system and the model server. For 8192 token context, the cache go add about 1 GB, so 8 GB plan go work but 4 GB plan no go work. If you want the full 128k context wey the model card advertise, the cache alone na 16 GB, so you dey look at 32 GB plan.

Why my model slow even though the VPS get plenty vCPUs?

Because generation dey limited by memory bandwidth, no be by cores. Every token need pull the whole active weight set from RAM, so once some cores saturate the memory channels, the remaining ones just dey wait. Swap na another common cause. If vmstat 1 show non zero si and so while the model dey answer, the weights no fit inside RAM. Part of every token dey come from disk, and that one cost much more than e suppose.

Longer context window really need more memory?

Yes, and the increase dey linear with tokens. A typical 8B model dey use about 128 KiB of KV cache per token, so 8192 tokens cost 1 GB and 131072 tokens cost 16 GB. The cache dey allocate when the model load, no be when the conversation dey grow. So if you request 128k context, that memory reserve immediately, even if every prompt wey you send na 200 tokens long.

Make I run large model at 2 bits or smaller one at 4 bits?

Choose the smaller model at 4 bits. Quality dey drop slowly from 8 bits to 4 bits, but e dey drop quickly below 4 bits. So 70B squeezed to 2 bits usually give worse answers than 32B at 4 bits from the same model generation. Heavy quantisation dey show as repetition and missed instructions, not as error message, so e easy to blame your prompt. Treat 4 bits as the minimum, then change the parameter count instead.

I fit self-host model wey get the same capability as big commercial ones?

No be for ordinary VPS. The strongest open weight models fit reach hundreds of billions of parameters. At 4 bits, that means more than 200 GB of RAM before any KV cache, and the strongest commercial models no dey distributed at all. Ordinary hardware dey do well when e run good 8B to 32B model for one specific job. For that kind work, narrow small model wey get good prompt often fit match general model. If you need frontier quality, compare the API price with the hardware cost before you buy either one.