VPS with GPU: when you actually need one

Do you need a VPS with GPU, or is CPU enough?

A VPS with GPU changes two things about running a model yourself: how fast tokens come out, and how large a model fits in memory at all. It changes nothing else. If your workload is a quantized 7B to 27B chat model answering one person at a time, an embedding job at low volume, or speech transcription with Whisper small, an ordinary CPU VPS with enough RAM already does the work. Start on CPU, measure the number that annoys you, then move up.

The reason is memory bandwidth. When a language model generates one token, it reads every weight it needs from memory. An 8B model quantized to 4 bits is roughly 4.7 GB on disk and about the same in memory, so producing a token means moving about 4.7 GB. Divide the memory bandwidth of the machine by that number and you have the ceiling on tokens per second. That single division explains almost every benchmark you will read.

What a GPU actually buys you

Bandwidth. Server DDR5 on a modern host moves tens of gigabytes per second. GPU memory (VRAM, video RAM) moves hundreds to over a thousand. The ratio is the speedup, and it is large.

Capacity with speed. A CPU box with 64 GB of RAM can load a 70B model at 4 bits. It will run, at a pace closer to reading than to chatting. A GPU only helps here if the model fits in VRAM, because the moment layers spill to system RAM the slow path is back in charge.

Batch throughput. This is the part people underestimate. A GPU generating for one user leaves most of its compute idle, since it is waiting on memory. Serve 20 requests at once and the same weight read serves all 20. Aggregate tokens per second rises several times over while per-user speed barely drops. A CPU does not do this. Two concurrent users on a CPU box roughly halve each other. If you are building an API that many clients call, batching is the argument for a GPU, more than raw single-stream speed.

Prompt processing. Reading a long prompt is compute-bound, not memory-bound, and this is where GPUs win by the widest margin. A 30,000 token context that a CPU chews through in a minute is a few seconds on a GPU. Retrieval setups that stuff documents into every request feel this constantly.

Rough numbers, and how to read them

The block below holds typical published single-stream figures for an 8B model at 4-bit quantization, as of July 2026. They are order-of-magnitude guidance, not a promise. Your quantization, context length and inference engine will move them.

[
  {
    "label": "8 vCPU, DDR4",
    "mem_bandwidth_gbs": 40,
    "tokens_per_sec": 6
  },
  {
    "label": "16 vCPU, DDR5",
    "mem_bandwidth_gbs": 75,
    "tokens_per_sec": 11
  },
  {
    "label": "24GB GPU",
    "mem_bandwidth_gbs": 300,
    "tokens_per_sec": 50
  },
  {
    "label": "40GB data-centre GPU",
    "mem_bandwidth_gbs": 1555,
    "tokens_per_sec": 130
  }
]

The 24 GB GPU row shows 50 tokens per second against 11 for a DDR5 CPU box. That is roughly five times, which tracks the bandwidth ratio rather than any difference in raw compute. Real throughput also lands below bandwidth divided by model size, because attention over a growing context adds work that the simple division ignores.

For comparison, a person reads at about 5 to 10 words per second. Anything at or above 15 tokens per second already feels like normal typing to a single reader. That is why so many CPU-only setups are quietly fine.

Sizing VRAM before you buy

Model file size is the floor, not the requirement. Budget the weights, plus the KV cache (key-value cache, the per-token memory that attention keeps), plus about 1 GB of overhead.

A practical rule as of July 2026: take the model file size in gigabytes and add 20 percent for a normal 8k to 16k context. A 4.7 GB 8B model wants about 6 GB of VRAM. A 27B model at 4 bits is around 16 GB and wants roughly 20 GB. A 70B at 4 bits is about 40 GB and needs a 48 GB card, or two smaller cards.

Long contexts break this rule. The KV cache grows linearly with context length, and at 128k tokens it can exceed the weights themselves. If you plan to use long contexts, size for the cache first and check what your engine offers for cache quantization.

Check what the machine actually has

On a GPU instance, confirm the driver sees the card before anything else.

nvidia-smi

You want a table listing the GPU name, the driver version, and memory used out of total. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver means the driver is missing, or the kernel module did not rebuild after a kernel upgrade. On a stock Ubuntu image the fix is usually sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers install, then a reboot so the new module loads.

For containers, the driver alone is not enough. Docker needs the NVIDIA Container Toolkit to pass the device through.

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then prove the passthrough works from inside a container:

sudo docker run --rm --gpus all ubuntu:24.04 nvidia-smi

The same table should appear. A docker: Error response from daemon: could not select device driver line, naming a gpu capability it cannot satisfy, means the toolkit is installed but Docker was never reconfigured or restarted, so re-run the nvidia-ctk line and the restart. In Compose the equivalent is a deploy.resources.reservations.devices entry whose driver is nvidia and whose capabilities list holds gpu, which slots into the ordinary service definitions covered in Docker Compose on a VPS.

Measure before you upgrade

Run the model you actually intend to use, on the CPU box you already have, and record the numbers. With Ollama self-hosting an LLM on a VPS this takes one flag:

ollama run llama3.1:8b --verbose "Summarise the causes of the 1929 crash in 200 words."

The output ends with timings. eval rate is your generation speed in tokens per second. prompt eval rate is how fast the machine read your input. Those two numbers tell you which upgrade helps: a low eval rate is a memory-bandwidth problem, and a low prompt eval rate on long inputs is a compute problem.

On a machine that has a GPU, check the model really landed on it:

ollama ps

The PROCESSOR column reads 100% GPU when everything fits, or something like 43%/57% CPU/GPU when it did not. A partial split is usually worse than you expect, because every token still waits on the slow half.

The cost question

GPU instances cost several times a comparable CPU instance, and they bill for every hour they exist, not for the tokens they produce. An always-on GPU serving a handful of requests a day is the most expensive way to run inference there is. The break-even is utilisation: a busy GPU is cheap per token, and an idle one is pure waste.

Three honest patterns work. Keep steady low-volume work on a CPU VPS. Send the occasional hard request to a hosted API and pay per token. Rent a GPU by the hour for batch jobs, fine-tuning, or a bulk embedding run, then destroy it. Mixing them is normal, and the budgeting discipline described in AI agent cost control on an always-on VPS applies here too, with the difference that idle time is the leak rather than token count.

What still runs fine without a GPU

Embeddings at low volume. A small embedding model processes hundreds of short documents per minute on a few CPU cores, and an index you build once does not need to be fast.

Whisper small and base for transcription. Faster-whisper on CPU transcribes near real time for the small model, which is enough for a pipeline that runs overnight.

Quantized chat models up to about 27B, for one or two users. Slow, readable, usable.

Anything you would call a batch job. If nobody is watching the screen, wall-clock speed is a scheduling detail rather than a requirement.

What genuinely needs a GPU: training or fine-tuning beyond a small adapter, serving many concurrent users, image and video generation, and real-time speech where latency is the product.

FAQ

How much VRAM do I need for a 7B or 8B model?

About 6 GB for a 4-bit quantized 8B model at a normal 8k to 16k context. The weights are roughly 4.7 GB, and the rest is the KV cache plus about 1 GB of overhead. A 12 GB card leaves comfortable room for longer contexts. If you plan to run at 128k context, size for the cache separately, because it can grow larger than the weights.

Can I run Ollama without a GPU?

Yes. Ollama falls back to CPU automatically and needs only enough RAM to hold the model. Expect roughly 5 to 12 tokens per second for a 4-bit 8B model depending on memory speed, which is close to reading pace for one user. Long prompts are the real pain point on CPU, because reading 30,000 tokens of context is compute-bound and takes far longer than generating the reply.

Why is my GPU barely faster than the CPU?

The usual cause is that the model did not fit entirely in VRAM, so some layers run on the CPU and every token waits on the slow half. Run ollama ps and check the PROCESSOR column reads 100% GPU. If it shows a split, use a smaller quantization or a smaller model. The other common cause is a short benchmark where model load time dominates the measurement.

Is a GPU VPS worth it for a single user?

Usually not. One person reads at 5 to 10 words per second, and a CPU box already produces tokens faster than that for models up to about 13B. The cases that justify the cost for a single user are long prompts, image generation, and fine-tuning. Serving many users at once is the strongest argument, because batching lets one GPU answer twenty requests for close to the cost of answering one.

Should I rent a GPU hourly or run one always on?

Rent hourly when the work is bursty: fine-tuning, a bulk embedding run, or a batch transcription job. Run always on only when the card stays busy, since a GPU instance bills for existing rather than for tokens produced. A low-traffic assistant is cheaper on a CPU VPS, or on a hosted API paid per token, than on an idle GPU.