VPS GPU: When You Really Need Am?

You need VPS wey get GPU, or CPU dey enough?

VPS wey get GPU dey change two things about running model by yourself: how fast tokens dey come out, and how large model fit enter memory at all. E no dey change anything else. If na quantized 7B to 27B chat model wey dey answer one person at a time, embedding job wey volume low, or speech transcription with Whisper small, ordinary CPU VPS wey get enough RAM don already fit do the work. Start with CPU, measure the number wey dey worry you, then move up.

The reason na memory bandwidth. When language model generate one token, e dey read every weight wey e need from memory. An 8B model quantized to 4 bits na roughly 4.7 GB for disk and about the same for memory, so to produce one token mean say machine go move about 4.7 GB. Divide the machine memory bandwidth by that number, and you get the limit for tokens per second. Na this one division explain almost every benchmark wey you go read.

GPU fiti actually dey give you

Bandwidth. Server DDR5 for modern host fit move tens of gigabytes per second. GPU memory (VRAM, video RAM) fit move hundreds reach pass one thousand. Na this ratio be the speedup, and e big.

Capacity with speed. CPU box wey get 64 GB RAM fit load 70B model for 4 bits. E go run, but the pace go near reading pass chatting. GPU only dey help here if the model fit enter VRAM, because once layers spill enter system RAM, the slow path don take control again.

Batch throughput. Na this part people dey underestimate. GPU wey dey generate for one user dey leave most of im compute idle, because e dey wait for memory. Serve 20 requests at once, and the same weight read go serve all 20. Total tokens per second go rise several times, while speed for each user go barely drop. CPU no dey do this. Two concurrent users for CPU box go roughly halve each other. If you dey build API wey many clients dey call, batching na the main reason to use GPU, more than raw single-stream speed.

Prompt processing. Reading long prompt na compute-bound work, no be memory-bound work, and na here GPU dey win by the widest margin. 30,000 token context wey CPU go chew through for one minute fit take few seconds for GPU. Retrieval setups wey dey put documents inside every request dey experience this all the time.

Rough figures, and how to read dem

The block wey dey below get typical published single-stream figures for an 8B model with 4-bit quantization, as of July 2026. Use dem as order-of-magnitude guidance, no be promise. Your quantization, context length and inference engine go change dem.

[
  {
    "label": "8 vCPU, DDR4",
    "mem_bandwidth_gbs": 40,
    "tokens_per_sec": 6
  },
  {
    "label": "16 vCPU, DDR5",
    "mem_bandwidth_gbs": 75,
    "tokens_per_sec": 11
  },
  {
    "label": "24GB GPU",
    "mem_bandwidth_gbs": 300,
    "tokens_per_sec": 50
  },
  {
    "label": "40GB data-centre GPU",
    "mem_bandwidth_gbs": 1555,
    "tokens_per_sec": 130
  }
]

The 24 GB GPU row dey show 50 tokens per second, compared with 11 for a DDR5 CPU box. Na roughly five times faster. This follow the bandwidth ratio, no be difference for raw compute. Real throughput still dey below bandwidth divided by model size, because attention over growing context dey add work wey the simple division no dey count.

For comparison, person dey read around 5 to 10 words per second. Anything wey reach 15 tokens per second or pass am already dey feel like normal typing for one reader. Na why plenty CPU-only setups dey work fine without much noise.

VRAM size check before you buy

Model file size na the minimum, e no be the full requirement. Plan for the weights, plus the KV cache (key-value cache, na the memory wey attention dey keep for each token), plus about 1 GB overhead.

Practical rule as of July 2026: take the model file size for gigabytes and add 20 percent for normal 8k to 16k context. 4.7 GB 8B model need about 6 GB VRAM. 27B model for 4 bits na around 16 GB and need roughly 20 GB. 70B for 4 bits na about 40 GB and need 48 GB card, or two smaller cards.

Long contexts fit break this rule. KV cache dey grow linearly with context length, and for 128k tokens e fit pass the weights size. If you plan to use long contexts, size for the cache first and check wetin your engine get for cache quantization.

Check wetin dey on the machine

For GPU instance, first confirm say the driver dey see the card before you do anything else.

nvidia-smi

You suppose see table wey list GPU name, driver version, and memory wey dem don use out of the total memory. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver mean say the driver no dey installed, or the kernel module no rebuild after kernel upgrade. For stock Ubuntu image, the usual fix na sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers install, then reboot so the new module go load.

For containers, driver alone no enough. Docker need NVIDIA Container Toolkit to pass the device through.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then prove say passthrough dey work from inside container:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi

The same table suppose show. A docker: Error response from daemon: could not select device driver line wey name a gpu capability wey e no fit satisfy mean say toolkit dey installed, but Docker never reconfigure or restart. So run nvidia-ctk line again and restart. For Compose, the equivalent na a deploy.resources.reservations.devices entry wey get driver as nvidia and capability list wey contain gpu. You fit add am to the ordinary service definitions wey Docker Compose for VPS cover.

Upgrade na first measure

Run the model wey you really plan to use, for the CPU machine wey you already get, and write down the numbers. With Ollama self-hosting an LLM for VPS this one flag dey enough:

ollama run llama3.1:8b --verbose "Summarise the causes of the 1929 crash in 200 words."

The output dey end with timing information. eval rate na your generation speed for tokens per second. prompt eval rate na how fast the machine read your input. These two numbers show which upgrade go help: low eval rate mean say memory bandwidth na the problem, while low prompt eval rate for long inputs mean say compute na the problem.

For machine wey get GPU, check say the model really load for am:

ollama ps

The PROCESSOR column go show 100% GPU when everything fit, or something like 43%/57% CPU/GPU when e no fit. Partial split usually worse pass wetin you expect, because every token still dey wait for the slow side.

The cost question

GPU instances cost several times wetin comparable CPU instance cost, and dem charge you for every hour wey dem dey exist, no be for the tokens wey dem produce. GPU wey dey always on and dey serve just small number of requests every day na the most expensive way to run inference. The break-even point na utilisation: busy GPU dey cheap per token, but idle one na pure waste.

Three honest patterns dey work. Keep steady low-volume work for CPU VPS. Send occasional hard request go hosted API and pay per token. Rent GPU by the hour for batch jobs, fine-tuning, or bulk embedding run, then destroy am. Mixing dem normal, and the budgeting discipline wey dem describe for AI agent cost control for always-on VPS apply here too. The difference be say idle time na the leak, instead of token count.

Wetin still dey run fine without GPU

Embeddings for small volume. Small embedding model fit process hundreds of short documents per minute with a few CPU cores. Index wey you build once no need to fast.

Whisper small and base for transcription. Faster-whisper for CPU fit transcribe near real time with the small model. This dey enough for pipeline wey run overnight.

Quantized chat models up to about 27B, for one or two users. E slow, but e readable and usable.

Anything wey you fit call batch job. If nobody dey watch the screen, wall-clock speed na scheduling detail, no be requirement.

Wetin genuinely need GPU: training or fine-tuning beyond small adapter, serving many concurrent users, image and video generation, and real-time speech where latency na the product.

FAQ

How much VRAM I need for a 7B or 8B model?

About 6 GB for a 4-bit quantized 8B model with normal 8k to 16k context. The weights na roughly 4.7 GB, while the rest na KV cache plus about 1 GB overhead. A 12 GB card go leave comfortable space for longer contexts. If you plan to run 128k context, calculate space for the cache separately, because e fit grow pass the weights.

I fit run Ollama without GPU?

Yes. Ollama go fall back to CPU automatically, and e only need enough RAM to hold the model. Expect roughly 5 to 12 tokens per second for 4-bit 8B model, depending on memory speed. This one dey close to reading speed for one user. Long prompts na the real problem for CPU, because reading 30,000 tokens of context na compute-bound and e dey take much longer than generating the reply.

Why my GPU barely faster than the CPU?

The common cause be say the model no fit enter VRAM completely. So, some layers dey run for CPU, and every token dey wait for the slower part. Run ollama ps and check whether the PROCESSOR column dey show 100% GPU. If e show split, use smaller quantization or smaller model. Another common cause na short benchmark where model load time dey make up most of the measurement.

GPU VPS worth am for one user?

Usually no. One person dey read 5 to 10 words per second, and CPU box already dey produce tokens faster than that for models up to about 13B. The situations wey fit justify the cost for one user na long prompts, image generation, and fine-tuning. Serving many users at the same time na the strongest reason, because batching lets one GPU answer twenty requests for almost the cost of answering one.

Make I rent GPU hourly or run one always on?

Rent am hourly when the work dey come in bursts: fine-tuning, bulk embedding run, or batch transcription job. Run am always on only when the card dey busy regularly, because GPU instance billing dey charge for the time e exists, not for the tokens e produces. Low-traffic assistant cheaper on CPU VPS, or hosted API wey you pay per token, than on idle GPU.