VPS GPU: When You Really Need Am One

You need VPS wey get GPU, or CPU go do?

VPS wey get GPU dey change two things when you dey run model by yourself: how fast tokens dey come out, and how big model fit enter memory at all. E no dey change anything else. If your workload na quantized 7B to 27B chat model wey dey answer one person at a time, embedding job for low volume, or speech transcription with Whisper small, ordinary CPU VPS wey get enough RAM fit already do the work. Start with CPU, measure the number wey dey disturb you, then move up.

The reason na memory bandwidth. When language model dey generate one token, e dey read every weight wey e need from memory. 8B model wey quantized to 4 bits na roughly 4.9 GB for disk and about the same for memory, so producing one token mean say machine dey move about 4.9 GB. Divide the machine memory bandwidth by that number, and you get the ceiling for tokens per second. That one division dey explain almost every benchmark wey you go read.

Wetin GPU really give you

Bandwidth. Server DDR5 for modern host fit move tens of gigabytes per second. GPU memory (VRAM, video RAM) fit move hundreds to over one thousand. Na this ratio be the speedup, and e big.

Capacity with speed. CPU box wey get 64 GB RAM fit load 70B model for 4 bits. E go run, but e pace go near reading pass chatting. GPU only help for here if the model fit enter VRAM, because once layers spill enter system RAM, slow path don take control again.

Batch throughput. Na this part people dey underestimate. GPU wey dey generate for one user leave most of its compute idle, because e dey wait for memory. Serve 20 requests at once, and the same weight read go serve all 20. Aggregate tokens per second go rise several times, while speed for each user go hardly drop. CPU inference engines like llama.cpp too support batching, and the gains depend on the hardware and workload. If you dey build API wey plenty clients dey call, batching na the main reason to use GPU, more than raw single-stream speed.

Prompt processing. Reading long prompt na compute-bound work, e no be memory-bound, and na here GPUs win by the widest margin. A 30,000 token context wey CPU dey process for one minute fit take few seconds for GPU. Retrieval setups wey dey put documents inside every request go make you notice this all the time.

Rough numbers, and how to read dem

Real throughput still dey below bandwidth divided by model size, because attention over context wey dey grow adds extra work wey the simple division no include.

For comparison, adult silent reading for English average about 4.0 to 4.3 words per second: 238 words per minute for non-fiction and 260 for fiction. Anything wey reach or pass 15 tokens per second already dey feel like normal typing for one reader. Na why plenty CPU-only setups dey quietly work fine.

Size VRAM before you buy

Model file size na the floor, e no be the full requirement. Budget the weights, plus the KV cache (key-value cache, the memory wey attention dey keep for every token), plus about 1 GB overhead.

Work out the cache instead of adding flat percentage, because quantizing the weights no dey quantize the cache. For every token, full-attention model dey store one key and one value for every layer and every KV head, at 2 bytes for each number in FP16, wey be Ollama default cache type. Ollama model metadata, wey dem check on 2026-10-27, give Llama 3.1 8B 32 layers and 8 KV heads of 128 dimensions. This one work out to 128 KiB per token: about 1.07 GB for 8k context and 2.15 GB for 16k. So 4.9 GB 8B model need about 7 GB VRAM for 8k and about 8 GB for 16k. 27B model for 4 bits dey around 17 GB (Gemma 3 27B for Ollama library). Gemma 3 metadata list 1,024-token sliding window, wey make plenty of its cache stop growing, so plan for 20 GB or more. 70B for 4 bits na about 43 GB (Llama 3.3 70B). With 80 layers, its FP16 cache add about 2.7 GB for 8k and 5.4 GB for 16k, so e need 48 GB card with small room left, or two smaller cards. This same arithmetic still dey work far beyond that, and the VRAM calculation for 2.8 trillion parameter model like Kimi K3 show where choosing a card no longer be the main question.

Long contexts na where this matter pass. For full-attention models like Llama 3.1, KV cache dey grow linearly with context length, and for 128k tokens e fit pass the weights themselves. Models with sliding-window or chunked attention dey different: Hugging Face Transformers documentation name Mistral, Gemma 2 and Llama 4 as examples where cache for those layers stop growing once e reach the window or chunk size, so their totals dey grow more slowly. If you plan to use long contexts, size for the cache first and check wetin your engine offer for cache quantization. For Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 dey use about half the memory wey default f16 dey use, and cache quantization only dey work when Flash Attention dey enabled. Set the context length explicitly too. Ollama dey pick its default from VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k for 48 GiB and above. For 48 GB card wey dey run 70B model, that default dey ask for far more cache than the card fit hold.

Check wetin the machine actually get

For a GPU instance, confirm say the driver dey see the card before you do anything else.

nvidia-smi

You suppose see table wey list the GPU name, driver version, and memory wey don use out of the total. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver mean say NVIDIA-SMI no fit reach the driver; check say you install the driver and say e kernel module don load. For a stock Ubuntu image, the usual fix na sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers install, then reboot so the new module fit load. For compute machines, Ubuntu server documentation recommend sudo ubuntu-drivers install --gpgpu, wey go select the server driver branch.

For containers, driver alone no dey enough. Docker need NVIDIA Container Toolkit to pass the device through.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then confirm say passthrough dey work from inside a container:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi

The same table suppose show. A docker: Error response from daemon: could not select device driver line, wey name a GPU capability wey e no fit satisfy, mean say Docker no fit find device driver wey match the request; check say you install the toolkit, then run the nvidia-ctk line and the restart again. For Compose, the equivalent na a deploy.resources.reservations.devices entry wey e driver be nvidia and whose capabilities list get gpu, wey fit enter the normal service definitions wey Docker Compose for VPS cover.

Measure before you upgrade

Run the model wey you really wan use for the CPU box wey you already get, then record the numbers. With Ollama self-hosting one LLM for VPS, na one flag dey needed:

ollama run llama3.1:8b --verbose "Summarise the causes of the 1929 crash in 200 words."

The output go end with timings. eval rate na your generation speed for tokens per second. prompt eval rate show how fast the machine read your input. These two numbers go tell you which upgrade go help: if eval rate low, na memory-bandwidth problem; if prompt eval rate low for long inputs, na compute problem.

For machine wey get GPU, check say the model really land for am:

ollama ps

The PROCESSOR column go show 100% GPU when everything fit, or something like 43%/57% CPU/GPU when e no fit. Partial split usually worse pass wetin you expect, because every token still dey wait for the slow half.

Cost matter

GPU instances dey bill based on compute time wey dem provision, no be token count. Provider dey set the rates and charges for stopped instances. GPU wey dey always on but dey serve only small number of requests each day go spend most of the paid time idle. The break-even point na utilisation: busy GPU cheap per token, but idle GPU na pure waste.

Three honest patterns dey work. Keep steady low-volume work for CPU VPS. Send occasional hard request go hosted API and pay per token. Rent GPU by the hour for batch jobs, fine-tuning, or bulk embedding run, then destroy am. To mix these options na normal. The budgeting discipline wey AI agent cost control for always-on VPS describe still apply here. The difference be say idle time na the leak, no be token count.

Wetin still dey run well without GPU

Embeddings for low volume. Small embedding model fit process hundreds of short documents per minute with few CPU cores, and index wey you build once no need make e fast.

Whisper small and base for transcription. Faster-whisper for CPU fit transcribe almost real time with small model, and this enough for pipeline wey dey run overnight.

Anything wey you go call batch job. If nobody dey watch the screen, wall-clock speed na scheduling detail, no be requirement.

Wetin genuinely need GPU: training or fine-tuning wey pass small adapter, serving plenty concurrent users, image and video generation, and real-time speech where latency na the product.

FAQ

How much VRAM I need for a 7B or 8B model?

About 7 GB for a 4-bit quantized 8B model with 8k context, and about 8 GB with 16k, when you dey use Ollama default FP16 cache. The weights dey around 4.9 GB. The KV cache add about 1.07 GB for 8k or 2.15 GB for 16k. The remaining overhead na about 1 GB. An 8 GB card fit hold am for 8k, but e go tight for 16k. A 12 GB card leave comfortable space for longer context. If you switch cache to q8_0, the cache share go reduce by about half. If you plan run 128k context, size the cache separately, because e fit grow pass the weights.

I fit run Ollama without GPU?

Yes. Ollama go automatically fall back to CPU, and e only need enough RAM to hold the model. Expect roughly 5 to 12 tokens per second for a 4-bit 8B model, depending on memory speed. For one user, this dey close to normal reading pace. Long prompts na the main problem for CPU, because processing 30,000 tokens of context dey compute-bound and e take much longer than generating the reply.

Why my GPU barely faster pass the CPU?

The usual cause na say the model no fit enter VRAM completely. So, some layers dey run for CPU, and every token dey wait for the slower part. Run ollama ps and check say the PROCESSOR column dey read 100% GPU. If e show split, use smaller quantization or smaller model. Another common cause na short benchmark where model load time dey dominate the measurement.

GPU VPS worth am for one user?

Usually, no. Adult silent reading in English dey average about 4.0 to 4.3 words per second. You must convert tokens per second to words per second before you compare CPU box speed with that pace. The cases wey fit justify the cost for one user na long prompts, image generation, and fine-tuning. Serving many users at the same time na the strongest reason, because batching allow one GPU share weight reads across requests.

I suppose rent GPU hourly or run one always on?

Rent am hourly when the work dey happen in bursts, like fine-tuning, bulk embedding run, or batch transcription job. Run am always on only when the card dey busy most of the time, because provisioned GPU compute billing dey based on time, not tokens produced. Low-traffic assistant go cost less on CPU VPS, or hosted API wey you pay per token, than on idle GPU.