VPS with GPU: when you actually need one

Do you need a VPS with GPU, or is CPU enough?

A VPS with GPU changes two things about running a model yourself: how fast tokens come out, and how large a model fits in memory at all. It changes nothing else. If your workload is a quantized 7B to 27B chat model answering one person at a time, an embedding job at low volume, or speech transcription with Whisper small, an ordinary CPU VPS with enough RAM already does the work. Start on CPU, measure the number that annoys you, then move up.

The reason is memory bandwidth. When a language model generates one token, it reads every weight it needs from memory. An 8B model quantized to 4 bits is roughly 4.9 GB on disk and about the same in memory, so producing a token means moving about 4.9 GB. Divide the memory bandwidth of the machine by that number and you have the ceiling on tokens per second. That single division explains almost every benchmark you will read.

What a GPU actually buys you

Bandwidth. Server DDR5 on a modern host moves tens of gigabytes per second. GPU memory (VRAM, video RAM) moves hundreds to over a thousand. The ratio is the speedup, and it is large.

Capacity with speed. A CPU box with 64 GB of RAM can load a 70B model at 4 bits. It will run, at a pace closer to reading than to chatting. A GPU only helps here if the model fits in VRAM, because the moment layers spill to system RAM the slow path is back in charge.

Batch throughput. This is the part people underestimate. A GPU generating for one user leaves most of its compute idle, since it is waiting on memory. Serve 20 requests at once and the same weight read serves all 20. Aggregate tokens per second rises several times over while per-user speed barely drops. CPU inference engines such as llama.cpp also support batching, with gains depending on the hardware and workload. If you are building an API that many clients call, batching is the argument for a GPU, more than raw single-stream speed.

Prompt processing. Reading a long prompt is compute-bound, not memory-bound, and this is where GPUs win by the widest margin. A 30,000 token context that a CPU chews through in a minute is a few seconds on a GPU. Retrieval setups that stuff documents into every request feel this constantly.

Rough numbers, and how to read them

Real throughput also lands below bandwidth divided by model size, because attention over a growing context adds work that the simple division ignores.

For comparison, adult silent reading in English averages about 4.0 to 4.3 words per second: 238 words per minute for non-fiction and 260 for fiction. Anything at or above 15 tokens per second already feels like normal typing to a single reader. That is why so many CPU-only setups are quietly fine.

Sizing VRAM before you buy

Model file size is the floor, not the requirement. Budget the weights, plus the KV cache (key-value cache, the per-token memory that attention keeps), plus about 1 GB of overhead.

Work the cache out rather than adding a flat percentage, because quantizing the weights does not quantize the cache. For each token, a full-attention model stores a key and a value for every layer and every KV head, at 2 bytes per number in FP16, which is Ollama's default cache type. Ollama's model metadata, checked on 2026-10-27, gives Llama 3.1 8B 32 layers and 8 KV heads of 128 dimensions, which works out to 128 KiB per token: about 1.07 GB at an 8k context and 2.15 GB at 16k. So a 4.9 GB 8B model wants about 7 GB of VRAM at 8k and about 8 GB at 16k. A 27B model at 4 bits is around 17 GB (Gemma 3 27B in Ollama's library). Gemma 3's metadata lists a 1,024-token sliding window, which keeps much of its cache from growing, so plan on 20 GB or more. A 70B at 4 bits is about 43 GB (Llama 3.3 70B). With 80 layers, its FP16 cache adds about 2.7 GB at 8k and 5.4 GB at 16k, so it needs a 48 GB card with little to spare, or two smaller cards. The same arithmetic keeps working far past that, and the VRAM math for a 2.8 trillion parameter model like Kimi K3 shows where picking a card stops being the question at all.

Long contexts are where this matters most. On full-attention models like Llama 3.1, the KV cache grows linearly with context length, and at 128k tokens it can exceed the weights themselves. Models with sliding-window or chunked attention are different: Hugging Face's Transformers documentation names Mistral, Gemma 2 and Llama 4 as examples where the cache on those layers stops growing once it reaches the window or chunk size, so their totals grow more slowly. If you plan to use long contexts, size for the cache first and check what your engine offers for cache quantization. In Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of the default f16, and cache quantization only applies when Flash Attention is enabled. Set the context length explicitly too. Ollama picks its default from VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB and above. On a 48 GB card running a 70B model, that default asks for far more cache than the card holds.

Check what the machine actually has

On a GPU instance, confirm the driver sees the card before anything else.

nvidia-smi

You want a table listing the GPU name, the driver version, and memory used out of total. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver means NVIDIA-SMI cannot reach the driver; check that the driver is installed and its kernel module is loaded. On a stock Ubuntu image the fix is usually sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers install, then a reboot so the new module loads. For compute machines, Ubuntu's server documentation recommends sudo ubuntu-drivers install --gpgpu, which picks the server driver branch.

For containers, the driver alone is not enough. Docker needs the NVIDIA Container Toolkit to pass the device through.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then prove the passthrough works from inside a container:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi

The same table should appear. A docker: Error response from daemon: could not select device driver line, naming a gpu capability it cannot satisfy, means Docker could not find a device driver matching the request; check that the toolkit is installed, then re-run the nvidia-ctk line and the restart. In Compose the equivalent is a deploy.resources.reservations.devices entry whose driver is nvidia and whose capabilities list holds gpu, which slots into the ordinary service definitions covered in Docker Compose on a VPS.

Measure before you upgrade

Run the model you actually intend to use, on the CPU box you already have, and record the numbers. With Ollama self-hosting an LLM on a VPS this takes one flag:

ollama run llama3.1:8b --verbose "Summarise the causes of the 1929 crash in 200 words."

The output ends with timings. eval rate is your generation speed in tokens per second. prompt eval rate is how fast the machine read your input. Those two numbers tell you which upgrade helps: a low eval rate is a memory-bandwidth problem, and a low prompt eval rate on long inputs is a compute problem.

On a machine that has a GPU, check the model really landed on it:

ollama ps

The PROCESSOR column reads 100% GPU when everything fits, or something like 43%/57% CPU/GPU when it did not. A partial split is usually worse than you expect, because every token still waits on the slow half.

The cost question

GPU instances bill for provisioned compute time rather than tokens, with rates and stopped-instance charges set by the provider. An always-on GPU serving a handful of requests a day spends most of its paid time idle. The break-even is utilisation: a busy GPU is cheap per token, and an idle one is pure waste.

Three honest patterns work. Keep steady low-volume work on a CPU VPS. Send the occasional hard request to a hosted API and pay per token. Rent a GPU by the hour for batch jobs, fine-tuning, or a bulk embedding run, then destroy it. Mixing them is normal, and the budgeting discipline described in AI agent cost control on an always-on VPS applies here too, with the difference that idle time is the leak rather than token count.

What still runs fine without a GPU

Embeddings at low volume. A small embedding model processes hundreds of short documents per minute on a few CPU cores, and an index you build once does not need to be fast.

Whisper small and base for transcription. Faster-whisper on CPU transcribes near real time for the small model, which is enough for a pipeline that runs overnight.

Quantized chat models up to about 27B, for one or two users. Slow, readable, usable.

Anything you would call a batch job. If nobody is watching the screen, wall-clock speed is a scheduling detail rather than a requirement.

What genuinely needs a GPU: training or fine-tuning beyond a small adapter, serving many concurrent users, image and video generation, and real-time speech where latency is the product.

FAQ

How much VRAM do I need for a 7B or 8B model?

About 7 GB for a 4-bit quantized 8B model at an 8k context, and about 8 GB at 16k, with Ollama's default FP16 cache. The weights are roughly 4.9 GB, the KV cache adds about 1.07 GB at 8k or 2.15 GB at 16k, and the rest is about 1 GB of overhead. An 8 GB card holds it at 8k and is tight at 16k, while a 12 GB card leaves comfortable room for longer contexts. Switching the cache to q8_0 roughly halves the cache's share. If you plan to run at 128k context, size for the cache separately, because it can grow larger than the weights.

Can I run Ollama without a GPU?

Yes. Ollama falls back to CPU automatically and needs only enough RAM to hold the model. Expect roughly 5 to 12 tokens per second for a 4-bit 8B model depending on memory speed, which is close to reading pace for one user. Long prompts are the real pain point on CPU, because reading 30,000 tokens of context is compute-bound and takes far longer than generating the reply.

Why is my GPU barely faster than the CPU?

The usual cause is that the model did not fit entirely in VRAM, so some layers run on the CPU and every token waits on the slow half. Run ollama ps and check the PROCESSOR column reads 100% GPU. If it shows a split, use a smaller quantization or a smaller model. The other common cause is a short benchmark where model load time dominates the measurement.

Is a GPU VPS worth it for a single user?

Usually not. Adult silent reading in English averages about 4.0 to 4.3 words per second, and tokens per second must be converted to words per second before comparing a CPU box with that pace. The cases that justify the cost for a single user are long prompts, image generation, and fine-tuning. Serving many users at once is the strongest argument, because batching lets one GPU share weight reads across requests.

Should I rent a GPU hourly or run one always on?

Rent hourly when the work is bursty: fine-tuning, a bulk embedding run, or a batch transcription job. Run always on only when the card stays busy, since provisioned GPU compute bills for time rather than tokens produced. A low-traffic assistant is cheaper on a CPU VPS, or on a hosted API paid per token, than on an idle GPU.