How to benchmark a VPS properly

What it means to benchmark a VPS

To benchmark a VPS you measure four things: how fast a single CPU core runs, how much memory bandwidth the machine has, how many small random disk operations the storage serves each second, and how much throughput the network link delivers. One run of yabs.sh gives you CPU, disk and network results in about ten minutes; use the separate memory test below for memory bandwidth. Reading the result is the harder part, because a VPS (virtual private server) shares physical hardware with other tenants, so the same machine can report one number at 03:00 and a very different one at 20:00.

The plan here is to run yabs.sh for a quick picture, then run the tools underneath it by hand. Running them yourself is what lets you change one flag, watch the number move, and learn what that number was really measuring. Do this after the machine is set up, not before. The steps in the first ten minutes on a new VPS come first, because a box still applying its first round of updates benchmarks badly for reasons that have nothing to do with the hardware.

Look at the machine before you measure it

Half of every bad benchmark is a machine the author did not understand.

nproc
lscpu | grep -E 'Model name|Hypervisor|Thread'
free -h
df -hT /
uname -r
systemd-detect-virt

Hypervisor vendor: KVM means full virtualisation, so you run your own kernel. systemd-detect-virt printing lxc or openvz means container virtualisation instead: you share the host kernel, and your CPU and memory limits are cgroup (control group) settings rather than virtual hardware. On a cgroup v2 system you can read the CPU limit from your container's cgroup; the following path applies when that cgroup is mounted at /sys/fs/cgroup.

cat /sys/fs/cgroup/cpu.max

max 100000 means this cgroup has no quota of its own; ancestor cgroups can still impose one. 200000 100000 means you may use 200000 microseconds of CPU in every 100000 microsecond period, which is two cores worth of quota. A plan advertised as 4 vCPU with a quota of two cores will never score like four cores. Benchmark tools can identify that restriction: VoxelBench reports CPU quotas and flags a shared or throttled CPU when the host exposes far more CPUs than the server may use.

df -hT / matters for a different reason: the Type column. If it reads overlay you are inside a container, and your disk results will describe whatever storage sits underneath that overlay. Note it now.

Watch steal time the whole time

Steal time is the share of time your virtual CPU spent waiting involuntarily while the hypervisor did not schedule it. It is a useful signal of unavailable CPU time, but it does not distinguish busy neighbours from the provider enforcing your CPU quota.

vmstat 1 10

Read the st column on the right. Sustained steal time means CPU time was unavailable to the guest, but a value above 5% does not prove the host is oversubscribed. Providers can deliberately expose CPU quota enforcement as steal time. top shows the same figure as %st on the CPU line. Keep vmstat 1 running in a second SSH session while you benchmark, and write the steal figure down next to each result.

Start with yabs.sh

yabs.sh (Yet Another Bench Script) is a shell script that runs fio, iperf3 and Geekbench and prints one summary. It uses fio and iperf3 if they are already installed, and downloads static binaries when they are not. It is the common language of VPS benchmark discussions, so a yabs output is the fastest way to compare notes with somebody else.

The project's own one-line form is this.

curl -sL yabs.sh | bash

That pipes whatever the URL serves today straight into a shell. Download it, read it, then run it.

curl -sLo yabs.sh https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh
less yabs.sh
bash yabs.sh

Flags go after -s -- when you pipe, or straight after the filename when you run a local copy. The useful ones: -f skips the disk test, -i skips the network test, -g skips Geekbench, -r cuts the iperf3 locations down to three, -j prints the results as JSON, and -w results.json writes that JSON to a file.

bash yabs.sh -r -w yabs-run1.json

Two things to know before the first run. Geekbench uploads your result and prints a public browser.geekbench.com URL, so anybody holding that link can read your CPU model and scores. -g skips that test completely. Second, the iperf3 stage moves real traffic to servers in several regions, and that counts against your monthly bandwidth allowance. On a 1 Gbit/s link a full network stage can move tens of gigabytes, so use -r on a small allowance and -i on a metered link.

What each part of the yabs output means

The disk section runs fio with a 50/50 read and write mix at four block sizes: 4k, 64k, 512k and 1m. It reports IOPS (input/output operations per second) and bandwidth for each one. The 4k row is the row to care about for a database, a mail server, or anything doing many small writes, because most server IO is small and scattered. The 1m row is the row for backups and video, where you move long runs of bytes.

The network section runs iperf3 against public servers in several regions, in both directions, using parallel streams. Treat a low number here as a question rather than an answer, because public iperf3 servers are shared and often saturated, so a poor result may belong to the far end.

The Geekbench section gives a single core score and a multi core score. Single core predicts how fast one request, one compile or one query finishes. Multi core mostly tells you how many cores you really got.

Disk: run fio yourself

fio (flexible IO tester) is the tool underneath the yabs disk section, and driving it directly is where the flags start to mean something.

sudo apt update && sudo apt install -y fio sysbench iperf3

A 4k random read test at queue depth 32, on the filesystem you actually care about:

fio --name=randread4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randread --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting

The summary line to read out of the output looks like this.

read: IOPS=184k, BW=719MiB/s (754MB/s)(42.1GiB/60001msec)

Under it fio prints a clat percentiles block. The 99.00th percentile is the figure worth quoting, because it is how long the slowest one request in a hundred waited. An average latency hides exactly the stalls a user notices.

• --direct=1 opens the file with O_DIRECT, so reads bypass the kernel page cache. Without it, the second pass over a 2G file on a machine with 8G of RAM is served from memory and fio reports IOPS in the millions. That number is real, and it is a memory number.
• --ioengine=libaio submits asynchronous requests, which is what lets --iodepth=32 keep 32 of them in flight. With a synchronous engine such as psync, an iodepth above 1 does nothing at all, so you measure one request at a time.
• --time_based --runtime=60 runs for a fixed 60 seconds instead of a fixed amount of work, so a fast disk and a slow disk get the same wall clock and the comparison stays fair.
• --size=2G sets the test file size. Keep it larger than any cache in the path, and check you have the free space first.

Random write is the same command with --rw=randwrite. Run it separately, then delete the file.

fio --name=randwrite4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randwrite --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting
rm -f ./fio-testfile

For a mix closer to real traffic, use --rw=randrw --rwmixread=70. Which class of storage you are sitting on changes these results by more than any flag does, and that split is covered in the difference between NVMe and SATA SSD storage on a VPS.

When fio stops with Unknown error -1

Direct IO is not available on every filesystem, so when fio fails on its very first request with --direct=1 set, direct IO is the first thing to rule out. The failure can look like this:

fio: io_u error on file ./fio-testfile: Unknown error -1: read offset=0, buflen=4096
fio: pid=1234, err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1

Run df -hT . first. A Type of overlay, the filesystem Docker gives a container by default, is not the cause on its own. overlayfs passes O_DIRECT through to the filesystem underneath and refuses it only when that filesystem cannot do direct IO, so --direct=1 --ioengine=libaio works on an overlay sitting on ext4 or xfs. When the filesystem under the test file really cannot do direct IO, fio usually says so as it opens the file: fio: looks like your file system does not support direct=1/buffered=0. In that case point --filename at a path on real local storage such as a bind mounted volume, or run fio on the host instead of in the container. If real storage is out of reach, a buffered synchronous run at least proves the command itself is correct.

fio --name=randread4k-buffered --filename=./fio-testfile --size=256M --bs=4k \
  --rw=randread --ioengine=psync --direct=0 --numjobs=1 \
  --runtime=15 --time_based --group_reporting
rm -f ./fio-testfile

Be honest about what that run is. After the first pass the 256M file sits in the page cache, so the IOPS figure describes your RAM. Use it to confirm fio is installed and the flags parse. Never quote it as a disk result.

Why dd is not a disk benchmark

dd appears in a lot of VPS threads, and it answers one narrow question.

dd if=/dev/zero of=./ddtest bs=1M count=1024 oflag=direct conv=fdatasync
rm -f ./ddtest

That measures sequential write throughput with one thread and one request in flight. It is a reasonable sanity check. It says nothing about random IO, and nothing about what happens when 32 requests arrive at once. Drop oflag=direct and writes go through the page cache, but conv=fdatasync still forces the output data to storage before the command finishes. Drop that too and the result can mostly measure how quickly your kernel accepts writes into memory, which is why dd figures quoted in forum posts are often absurd.

CPU: sysbench cpu

sysbench cpu --cpu-max-prime=20000 --threads=1 run
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

The figure to keep is events per second. Run single threaded first. That is the number deciding how quickly one PHP request finishes or one compile job completes, and it varies most between hosts at the same price. Then run with every thread, which shows how throughput scales across your allocated vCPUs.

Be clear about what this measures: sysbench cpu repeatedly finds prime numbers by trial division. For each candidate it takes a double precision square root, then tests divisors with 64 bit integer modulo, comparing every divisor against that floating point limit. So it mixes integer and floating point work. It does not stress memory bandwidth, vector units or cache in any way that resembles a real workload, so it is good for ranking two hosts and poor for predicting how your application will run.

Ubuntu 24.04 ships sysbench 1.0.20, where the test name comes first. Copy a command with --test=cpu from an old post and you get WARNING: the --test option is deprecated. Scores from sysbench 0.4 and sysbench 1.0 are not comparable at all, so never measure yourself against a published number that does not name its version.

Memory: sysbench memory

sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=write --threads=1 run
sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=read --threads=1 run

The result is in MiB/sec. Keep --memory-block-size at 1M, and keep it identical across every host you compare. At 1K the number collapses, because you pay the per-operation overhead a thousand times more often, so you end up measuring loop cost rather than memory bandwidth. This is the most commonly mismatched flag in published memory scores.

Network: iperf3

The honest way to test throughput is against a second machine you control, because then you know what both ends are doing.

On the far end:

iperf3 -s

That listens on TCP 5201. Open the port only for the address you test from, and close it when you finish. Basic ufw firewall rules on a VPS covers the syntax.

From the VPS under test:

iperf3 -c 203.0.113.10 -t 30
iperf3 -c 203.0.113.10 -t 30 -R
iperf3 -c 203.0.113.10 -t 30 -P 8

The first measures upload from the machine under test. -R reverses the direction, which measures download. -P 8 opens eight parallel streams.

Run the single stream and the parallel version both, because they answer different questions. One TCP connection can only hold as much unacknowledged data as its window allows, so its ceiling is roughly the window size divided by the round trip time. At 80 ms of latency with a 4 MB window that ceiling is about 400 Mbit/s, however fast the link underneath is. The single stream figure tells you what one download will get. The parallel figure tells you the capacity of the link.

Keep an eye on your bandwidth allowance while you do this. Thirty seconds at 1 Gbit/s moves about 3.75 GB, and you will run it several times in each direction.

Reference figures, and how to read yours

[
  {
    "device": "Local NVMe",
    "iops_4k_read": "180,000"
  },
  {
    "device": "Local SATA SSD",
    "iops_4k_read": "90,000"
  },
  {
    "device": "Network block",
    "iops_4k_read": "12,000"
  },
  {
    "device": "Spinning disk",
    "iops_4k_read": "180"
  }
]

A local NVMe volume in published results usually lands near 180,000 4k random read IOPS. A local SATA SSD comes in around 90,000. Network attached block storage, where every request crosses a network before it reaches a disk, sits nearer 12,000, and a spinning disk manages roughly 180, because it moves a physical head for every random request.

These are typical published figures for each class of storage, not measurements from one host. Use them for one purpose: checking that your own result is in the right order of magnitude. If a plan sold as NVMe benchmarks in the low thousands of 4k IOPS, first confirm --direct=1 was on. If it was, check the plan's IOPS limits and burst credits as well as storage contention and the hardware advertised.

Why one run is not a benchmark

A single result is a snapshot of one minute on a shared machine. Treat it as one sample.

• Run every test at least five times, spread across different hours and at least two different days. Keep the median and the spread. A result published without a spread is a marketing figure.
• Record the steal time beside each run. Keep runs where st was high and investigate whether they reflect contention or the plan's CPU quota; discarding them can hide the performance you actually receive.
• Run the disk test at two durations. Many plans give a burst IOPS allowance that refills over time, but neither a 60 second fio run nor --runtime=600 guarantees you have measured the floor. AWS gp2 credits can sustain 3,000 IOPS for at least 30 minutes from a full balance, so both runs can measure burst performance. Check the provider's credit model and monitor the credit balance before calling a result sustained performance.
• Check nothing else is running. unattended-upgrades starting an apt transaction in the middle of a CPU test costs you real points, and ps -e -o comm= | grep -E 'apt|dpkg' before each run takes a second.
• Change one variable at a time. Different tool versions, block sizes or thread counts produce numbers that cannot be compared, no matter how similar they look.

When you compare two providers, run them at the same hour of the same day. Otherwise you have measured the time of day.

Benchmark your own workload last

Synthetic tools rank machines. Only your own workload tells you whether a machine is enough. Time the thing you actually do.

time tar -czf /tmp/bench.tgz /usr/share
rm -f /tmp/bench.tgz

That compresses a few hundred megabytes, so it exercises CPU and disk together and moves when either one changes. The Removing leading / from member names warning is normal. Better still, time your own build, your own slowest query, or your own page render. A build taking 4 minutes on one host and 7 on another has settled the question, whatever Geekbench thought. This is also the measurement that tells you when more machine stops being worth paying for, which is worth knowing before you read what a VPS actually costs per month or move the workload onto a dedicated server.

FAQ

Why do I get a different benchmark result every time I run it?

A VPS shares physical CPU, storage and network with other tenants, so your result depends on what they are doing at that moment. Run vmstat 1 during the test and read the st column: sustained steal time records unavailable CPU time, but it can reflect the provider enforcing your purchased CPU allocation as well as host contention. A value above 5% does not establish the cause. The answer is method rather than tuning. Run each test five or more times across different hours, then report the median together with the spread.

Why does fio report millions of IOPS?

Almost always because --direct=1 is missing. Without it fio reads through the kernel page cache, so after the first pass a 2G test file is served from RAM and you have measured memory bandwidth. Add --direct=1 and keep the test file larger than any cache in the path. If --direct=1 then fails, run df -hT . and check what the test file sits on. An overlay type is not the problem by itself, because overlayfs passes O_DIRECT through to the storage underneath. Move the test file to real local storage only if the filesystem underneath refuses direct IO.

Is yabs.sh enough on its own?

For a first look, yes. It runs fio at four block sizes, iperf3 in both directions and Geekbench, and it prints one summary other people can read. It stops being enough when you want to know why a number is what it is, because you cannot vary its flags per test. Once a yabs result looks wrong, reproduce it with fio or sysbench directly and change one flag at a time.

Which single number predicts how my application will feel?

Single core CPU speed and 4k random read latency, in that order, for most web and database workloads. Throughput figures look impressive and rarely decide anything, because a typical request is small. Quote the 99th percentile from the fio clat percentiles block rather than the average, since the slow one request in a hundred is the one a user notices.

Do I need to install anything before benchmarking?

fio, sysbench and iperf3 are all in the Ubuntu and Debian archives: sudo apt install -y fio sysbench iperf3. yabs.sh runs in Bash and needs curl or wget for downloads, plus tar to extract Geekbench; without tar, it skips Geekbench. It uses fio and iperf3 when they are installed and downloads static binaries when they are missing. Delete every test file when you finish, since a 2G fio file left behind on a 20G disk becomes somebody's disk full alert weeks later.