VPS GPU: When You Really Need Am One

You need VPS wey get GPU, or CPU go dey enough?

VPS wey get GPU dey change two things when you dey run model by yourself: how fast tokens dey come out, and how big model fit enter memory at all. E no dey change anything else. If na quantized 7B to 27B chat model wey dey answer one person at a time, embedding job wey volume low, or speech transcription with Whisper small, ordinary CPU VPS wey RAM enough already fit do the work. Start with CPU, measure the number wey dey bother you, then move up.

The reason na memory bandwidth. When language model generate one token, e dey read every weight wey e need from memory. 8B model wey quantized to 4 bits dey roughly 4.7 GB for disk and about the same size for memory, so to produce one token means say machine dey move about 4.7 GB. Divide the machine memory bandwidth by that number, and you get the ceiling for tokens per second. That one division explain almost every benchmark wey you go read.

Wetin GPU really dey buy you

Bandwidth. Server DDR5 for modern host fit move tens of gigabytes per second. GPU memory (VRAM, video RAM) fit move hundreds reach pass one thousand. Na this ratio be the speedup, and e big.

Capacity with speed. CPU box wey get 64 GB RAM fit load 70B model for 4 bits. E go run, but e pace go near reading pass chatting. GPU only help here if model fit enter VRAM, because once layers spill enter system RAM, slow path don take charge again.

Batch throughput. Na this part people dey underestimate. GPU wey dey generate for one user leave most of e compute idle, because e dey wait for memory. Serve 20 requests at once, and the same weight read go serve all 20. Aggregate tokens per second go rise several times, while per-user speed barely drop. CPU no dey do this. Two concurrent users for CPU box go roughly halve each other speed. If you dey build API wey many clients dey call, batching na the main reason to use GPU, more than raw single-stream speed.

Prompt processing. Reading long prompt na compute-bound, no be memory-bound, and na here GPU win by the widest margin. 30,000 token context wey CPU go chew for one minute fit take few seconds for GPU. Retrieval setups wey dey stuff documents inside every request dey feel this all the time.

Rough numbers, and how to read dem

The block wey dey down get typical published single-stream figures for an 8B model for 4-bit quantization, as of July 2026. Dem na rough guidance for scale, no be promise. Your quantization, context length and inference engine go change dem.

[
  {
    "label": "8 vCPU, DDR4",
    "mem_bandwidth_gbs": 40,
    "tokens_per_sec": 6
  },
  {
    "label": "16 vCPU, DDR5",
    "mem_bandwidth_gbs": 75,
    "tokens_per_sec": 11
  },
  {
    "label": "24GB GPU",
    "mem_bandwidth_gbs": 300,
    "tokens_per_sec": 50
  },
  {
    "label": "40GB data-centre GPU",
    "mem_bandwidth_gbs": 1555,
    "tokens_per_sec": 130
  }
]

The 24 GB GPU row show 50 tokens per second, compared with 11 for a DDR5 CPU box. Na roughly five times faster. This match the bandwidth ratio more than any difference for raw compute. Actual throughput still dey below bandwidth divided by model size, because attention over context wey dey grow add more work wey the simple division no include.

For comparison, person dey read about 5 to 10 words per second. Anything wey reach 15 tokens per second or pass am already dey feel like normal typing for one reader. Na why plenty CPU-only setups dey work fine without much noise.

VRAM size wey you need before you buy

Model file size na the lowest requirement, no be the full requirement. Budget for the weights, plus the KV cache (key-value cache, na the memory wey attention dey keep for each token), plus about 1 GB overhead.

Practical rule as of July 2026: take the model file size for gigabytes and add 20 percent for normal 8k to 16k context. 4.7 GB 8B model need about 6 GB VRAM. 27B model for 4 bits dey around 16 GB and need roughly 20 GB. 70B for 4 bits dey about 40 GB and need 48 GB card, or two smaller cards. This same calculation still work well beyond that, and the VRAM calculation for 2.8 trillion parameter model like Kimi K3 show where choosing card no longer be the main question.

Long contexts break this rule. KV cache dey grow linearly with context length, and for 128k tokens e fit pass the weights themselves. If you plan to use long contexts, size for the cache first and check wetin your engine offer for cache quantization.

Check wetin the machine really get

For GPU instance, confirm say the driver dey see the card before you do anything else.

nvidia-smi

You want table wey list the GPU name, driver version, and memory wey dem don use out of the total. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver mean say the driver no dey, or the kernel module no rebuild after kernel upgrade. For standard Ubuntu image, the fix normally na sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers install, then reboot so the new module fit load.

For containers, driver alone no enough. Docker need NVIDIA Container Toolkit to pass the device through.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then prove say the passthrough dey work from inside container:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi

The same table suppose show. docker: Error response from daemon: could not select device driver line wey name a GPU capability wey e no fit satisfy mean say toolkit don install, but Docker never reconfigure or restart, so run nvidia-ctk line again and do the restart. For Compose, the equivalent na deploy.resources.reservations.devices entry wey e driver be nvidia and wey capabilities list get gpu, and e fit enter the normal service definitions wey Docker Compose for VPS cover.

Measure before you upgrade

Run the model wey you really wan use for the CPU box wey you already get, then record the numbers. With Ollama self-hosting one LLM for VPS, na one flag dey needed:

ollama run llama3.1:8b --verbose "Summarise the causes of the 1929 crash in 200 words."

The output go end with timings. eval rate na your generation speed for tokens per second. prompt eval rate show how fast the machine read your input. These two numbers go tell you which upgrade go help: if eval rate low, na memory-bandwidth problem; if prompt eval rate low for long inputs, na compute problem.

For machine wey get GPU, check say the model really land for am:

ollama ps

The PROCESSOR column go show 100% GPU when everything fit, or something like 43%/57% CPU/GPU when e no fit. Partial split usually worse pass wetin you expect, because every token still dey wait for the slow half.

Cost matter

GPU instances cost several times pass comparable CPU instance, and dem dey charge for every hour wey dem exist, no be for the tokens wey dem produce. GPU wey dey always on to serve just few requests per day na the most expensive way to run inference. The main break-even point na utilisation: busy GPU cheap per token, but idle one na pure waste.

Three honest patterns dey work. Keep steady low-volume work for CPU VPS. Send occasional hard request to hosted API and pay per token. Rent GPU by the hour for batch jobs, fine-tuning, or bulk embedding run, then destroy am. Mixing dem na normal, and the budgeting discipline wey dem describe for AI agent cost control for always-on VPS apply here too. The difference be say idle time na the leak, no be token count.

Wetin still dey run well without GPU

Embeddings for low volume. Small embedding model fit process hundreds of short documents per minute with few CPU cores, and index wey you build once no need make e fast.

Whisper small and base for transcription. Faster-whisper for CPU fit transcribe almost real time with small model, and this enough for pipeline wey dey run overnight.

Anything wey you go call batch job. If nobody dey watch the screen, wall-clock speed na scheduling detail, no be requirement.

Wetin genuinely need GPU: training or fine-tuning wey pass small adapter, serving plenty concurrent users, image and video generation, and real-time speech where latency na the product.

FAQ

7B or 8B model need how much VRAM?

About 6 GB for 4-bit quantized 8B model with normal 8k to 16k context. The weights na roughly 4.7 GB, while the remaining space na for KV cache plus about 1 GB overhead. 12 GB card go leave comfortable space for longer contexts. If you plan run am for 128k context, calculate the cache space separately, because e fit grow pass the weights.

I fit run Ollama without GPU?

Yes. Ollama go automatically fall back to CPU, and e need only enough RAM to hold the model. Expect roughly 5 to 12 tokens per second for 4-bit 8B model, depending on memory speed. For one user, this dey close to reading speed. Long prompts na the main problem for CPU, because reading 30,000 tokens of context na compute-bound and e dey take much longer than generating the reply.

Why my GPU barely pass CPU for speed?

The usual cause na say the model no fit enter VRAM completely, so some layers dey run for CPU and every token dey wait for the slower side. Run ollama ps and check whether the PROCESSOR column dey show 100% GPU. If e show split, use smaller quantization or smaller model. Another common cause na short benchmark where model load time dey dominate the measurement.

GPU VPS worth am for one user?

Usually no. One person dey read 5 to 10 words per second, and CPU box already dey produce tokens faster than that for models up to about 13B. Long prompts, image generation, and fine-tuning na the cases wey fit justify the cost for one user. Serving many users at once na the strongest reason, because batching allow one GPU answer twenty requests for almost the same cost as answering one.

Make I rent GPU hourly or run one always on?

Rent am hourly when the work dey come in bursts: fine-tuning, bulk embedding run, or batch transcription job. Run am always on only when the card dey busy most of the time, because GPU instance dey bill you for existing, not for the tokens e produce. Low-traffic assistant cheaper for CPU VPS, or hosted API wey you pay per token, than for idle GPU.