How to benchmark a VPS properly

What it means to benchmark a VPS

To benchmark a VPS you measure four things: how fast a single CPU core runs, how much memory bandwidth the machine has, how many small random disk operations the storage serves each second, and how much throughput the network link delivers. One run of yabs.sh gives you all four in about ten minutes. Reading the result is the harder part, because a VPS (virtual private server) shares physical hardware with other tenants, so the same machine can report one number at 03:00 and a very different one at 20:00.

The plan here is to run yabs.sh for a quick picture, then run the tools underneath it by hand. Running them yourself is what lets you change one flag, watch the number move, and learn what that number was really measuring. Do this after the machine is set up, not before. The steps in the first ten minutes on a new VPS come first, because a box still applying its first round of updates benchmarks badly for reasons that have nothing to do with the hardware.

Look at the machine before you measure it

Half of every bad benchmark is a machine the author did not understand.

nproc
lscpu | grep -E 'Model name|Hypervisor|Thread'
free -h
df -hT /
uname -r
systemd-detect-virt

Hypervisor vendor: KVM means full virtualisation, so you run your own kernel. systemd-detect-virt printing lxc or openvz means container virtualisation instead: you share the host kernel, and your CPU and memory limits are cgroup (control group) settings rather than virtual hardware. On a cgroup v2 system you can read the CPU limit directly.

cat /sys/fs/cgroup/cpu.max

max 100000 means there is no quota. 200000 100000 means you may use 200000 microseconds of CPU in every 100000 microsecond period, which is two cores worth of quota. A plan advertised as 4 vCPU with a quota of two cores will never score like four cores, and no benchmark tool prints a line telling you why.

df -hT / matters for a different reason: the Type column. If it reads overlay you are inside a container, and the disk test below needs a change. Note it now.

Watch steal time the whole time

Steal time is the share of time your virtual CPU was ready to run and the hypervisor handed the physical core to somebody else. It is the most useful single signal that a result is about your neighbours rather than about the hardware.

vmstat 1 10

Read the st column on the right. A steady 0 or 1 is normal. Sustained values above 5 mean the host is oversubscribed at that moment, so every CPU number you record in that window is low through no fault of your machine. top shows the same figure as %st on the CPU line. Keep vmstat 1 running in a second SSH session while you benchmark, and write the steal figure down next to each result.

Start with yabs.sh

yabs.sh (Yet Another Bench Script) is a shell script that downloads static fio, iperf3 and Geekbench binaries, runs them, and prints one summary. It is the common language of VPS benchmark discussions, so a yabs output is the fastest way to compare notes with somebody else.

The project's own one-line form is this.

curl -sL yabs.sh | bash

That pipes whatever the URL serves today straight into a shell. Download it, read it, then run it.

curl -sLo yabs.sh https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh
less yabs.sh
bash yabs.sh

Flags go after -s -- when you pipe, or straight after the filename when you run a local copy. The useful ones: -f skips the disk test, -i skips the network test, -g skips Geekbench, -r cuts the iperf3 locations down to two, -j prints the results as JSON, and -w results.json writes that JSON to a file.

bash yabs.sh -r -w yabs-run1.json

Two things to know before the first run. Geekbench uploads your result and prints a public browser.geekbench.com URL, so anybody holding that link can read your CPU model and scores. -g skips that test completely. Second, the iperf3 stage moves real traffic to servers in several regions, and that counts against your monthly bandwidth allowance. On a 1 Gbit/s link a full network stage can move tens of gigabytes, so use -r on a small allowance and -i on a metered link.

What each part of the yabs output means

The disk section runs fio with a 50/50 read and write mix at four block sizes: 4k, 64k, 512k and 1m. It reports IOPS (input/output operations per second) and bandwidth for each one. The 4k row is the row to care about for a database, a mail server, or anything doing many small writes, because most server IO is small and scattered. The 1m row is the row for backups and video, where you move long runs of bytes.

The network section runs iperf3 against public servers in several regions, in both directions, using parallel streams. Treat a low number here as a question rather than an answer, because public iperf3 servers are shared and often saturated, so a poor result may belong to the far end.

The Geekbench section gives a single core score and a multi core score. Single core predicts how fast one request, one compile or one query finishes. Multi core mostly tells you how many cores you really got.

Disk: run fio yourself

fio (flexible IO tester) is the tool underneath the yabs disk section, and driving it directly is where the flags start to mean something.

sudo apt update && sudo apt install -y fio sysbench iperf3

A 4k random read test at queue depth 32, on the filesystem you actually care about:

fio --name=randread4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randread --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting

The summary line to read out of the output looks like this.

read: IOPS=184k, BW=719MiB/s (754MB/s)(42.1GiB/60001msec)

Under it fio prints a clat percentiles block. The 99.00th percentile is the figure worth quoting, because it is how long the slowest one request in a hundred waited. An average latency hides exactly the stalls a user notices.

• --direct=1 opens the file with O_DIRECT, so reads bypass the kernel page cache. Without it, the second pass over a 2G file on a machine with 8G of RAM is served from memory and fio reports IOPS in the millions. That number is real, and it is a memory number.
• --ioengine=libaio submits asynchronous requests, which is what lets --iodepth=32 keep 32 of them in flight. With a synchronous engine such as psync, an iodepth above 1 does nothing at all, so you measure one request at a time.
• --time_based --runtime=60 runs for a fixed 60 seconds instead of a fixed amount of work, so a fast disk and a slow disk get the same wall clock and the comparison stays fair.
• --size=2G sets the test file size. Keep it larger than any cache in the path, and check you have the free space first.

Random write is the same command with --rw=randwrite. Run it separately, then delete the file.

fio --name=randwrite4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randwrite --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting
rm -f ./fio-testfile

For a mix closer to real traffic, use --rw=randrw --rwmixread=70. Which class of storage you are sitting on changes these results by more than any flag does, and that split is covered in the difference between NVMe and SATA SSD storage on a VPS.

When fio stops with Unknown error -1

Direct IO is not available on every filesystem. overlay, the filesystem Docker gives a container by default, and several network filesystems do not support O_DIRECT, so libaio submits a request the kernel cannot complete and fio gives up:

fio: io_u error on file ./fio-testfile: Unknown error -1: read offset=0, buflen=4096
fio: pid=1234, err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1

Run df -hT . first. If the Type column says overlay, point --filename at a path on real storage such as a bind mounted volume, or run fio on the host instead of in the container. If real storage is out of reach, a buffered synchronous run at least proves the command itself is correct.

fio --name=randread4k-buffered --filename=./fio-testfile --size=256M --bs=4k \
  --rw=randread --ioengine=psync --direct=0 --numjobs=1 \
  --runtime=15 --time_based --group_reporting
rm -f ./fio-testfile

Be honest about what that run is. After the first pass the 256M file sits in the page cache, so the IOPS figure describes your RAM. Use it to confirm fio is installed and the flags parse. Never quote it as a disk result.

Why dd is not a disk benchmark

dd appears in a lot of VPS threads, and it answers one narrow question.

dd if=/dev/zero of=./ddtest bs=1M count=1024 oflag=direct conv=fdatasync
rm -f ./ddtest

That measures sequential write throughput with one thread and one request in flight. It is a reasonable sanity check. It says nothing about random IO, and nothing about what happens when 32 requests arrive at once. Drop oflag=direct and it mostly measures how quickly your kernel accepts writes into memory, which is why dd figures quoted in forum posts are often absurd.

CPU: sysbench cpu

sysbench cpu --cpu-max-prime=20000 --threads=1 run
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

The figure to keep is events per second. Run single threaded first. That is the number deciding how quickly one PHP request finishes or one compile job completes, and it varies most between hosts at the same price. Then run with every thread, which shows whether your vCPUs are separate cores or slices of one.

Be clear about what this measures: sysbench cpu repeatedly finds prime numbers using 64 bit integer arithmetic. It does not stress memory bandwidth, vector units or cache in any way that resembles a real workload, so it is good for ranking two hosts and poor for predicting how your application will run.

Ubuntu 24.04 ships sysbench 1.0.20, where the test name comes first. Copy a command with --test=cpu from an old post and you get WARNING: the --test option is deprecated. Scores from sysbench 0.4 and sysbench 1.0 are not comparable at all, so never measure yourself against a published number that does not name its version.

Memory: sysbench memory

sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=write --threads=1 run
sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=read --threads=1 run

The result is in MiB/sec, and reads come out faster than writes on every machine. Keep --memory-block-size at 1M, and keep it identical across every host you compare. At 1K the number collapses, because you pay the per-operation overhead a thousand times more often, so you end up measuring loop cost rather than memory bandwidth. This is the most commonly mismatched flag in published memory scores.

Network: iperf3

The honest way to test throughput is against a second machine you control, because then you know what both ends are doing.

On the far end:

iperf3 -s

That listens on TCP 5201. Open the port only for the address you test from, and close it when you finish. Basic ufw firewall rules on a VPS covers the syntax.

From the VPS under test:

iperf3 -c 203.0.113.10 -t 30
iperf3 -c 203.0.113.10 -t 30 -R
iperf3 -c 203.0.113.10 -t 30 -P 8

The first measures upload from the machine under test. -R reverses the direction, which measures download. -P 8 opens eight parallel streams.

Run the single stream and the parallel version both, because they answer different questions. One TCP connection can only hold as much unacknowledged data as its window allows, so its ceiling is roughly the window size divided by the round trip time. At 80 ms of latency with a 4 MB window that ceiling is about 400 Mbit/s, however fast the link underneath is. The single stream figure tells you what one download will get. The parallel figure tells you the capacity of the link.

Keep an eye on your bandwidth allowance while you do this. Thirty seconds at 1 Gbit/s moves about 3.75 GB, and you will run it several times in each direction.

Reference figures, and how to read yours

[
  {
    "device": "Local NVMe",
    "iops_4k_read": "180,000"
  },
  {
    "device": "Local SATA SSD",
    "iops_4k_read": "90,000"
  },
  {
    "device": "Network block",
    "iops_4k_read": "12,000"
  },
  {
    "device": "Spinning disk",
    "iops_4k_read": "180"
  }
]

A local NVMe volume in published results usually lands near 180,000 4k random read IOPS. A local SATA SSD comes in around 90,000. Network attached block storage, where every request crosses a network before it reaches a disk, sits nearer 12,000, and a spinning disk manages roughly 180, because it moves a physical head for every random request.

These are typical published figures for each class of storage, not measurements from one host. Use them for one purpose: checking that your own result is in the right order of magnitude. If a plan sold as NVMe benchmarks in the low thousands of 4k IOPS, first confirm --direct=1 was on. If it was, then either the storage is not what the product page describes, or you are sharing it with a very busy neighbour.

Why one run is not a benchmark

A single result is a snapshot of one minute on a shared machine. Treat it as one sample.

• Run every test at least five times, spread across different hours and at least two different days. Keep the median and the spread. A result published without a spread is a marketing figure.
• Record the steal time beside each run. Throw out runs where st was high, or at minimum note that it was.
• Run the disk test at two durations. Many plans give a burst IOPS allowance that refills over time, so a 60 second fio run measures the burst while --runtime=600 measures the floor. The floor is what you get on a bad day.
• Check nothing else is running. unattended-upgrades starting an apt transaction in the middle of a CPU test costs you real points, and ps -e -o comm= | grep -E 'apt|dpkg' before each run takes a second.
• Change one variable at a time. Different tool versions, block sizes or thread counts produce numbers that cannot be compared, no matter how similar they look.

When you compare two providers, run them at the same hour of the same day. Otherwise you have measured the time of day.

Benchmark your own workload last

Synthetic tools rank machines. Only your own workload tells you whether a machine is enough. Time the thing you actually do.

time tar -czf /tmp/bench.tgz /usr/share
rm -f /tmp/bench.tgz

That compresses a few hundred megabytes, so it exercises CPU and disk together and moves when either one changes. The Removing leading / from member names warning is normal. Better still, time your own build, your own slowest query, or your own page render. A build taking 4 minutes on one host and 7 on another has settled the question, whatever Geekbench thought. This is also the measurement that tells you when more machine stops being worth paying for, which is worth knowing before you read what a VPS actually costs per month or move the workload onto a dedicated server.

FAQ

Why do I get a different benchmark result every time I run it?

A VPS shares physical CPU, storage and network with other tenants, so your result depends on what they are doing at that moment. Run vmstat 1 during the test and read the st column: sustained steal time above 5 means the host was busy, and your CPU score is low for reasons outside your machine. The answer is method rather than tuning. Run each test five or more times across different hours, then report the median together with the spread.

Why does fio report millions of IOPS?

Almost always because --direct=1 is missing. Without it fio reads through the kernel page cache, so after the first pass a 2G test file is served from RAM and you have measured memory bandwidth. Add --direct=1 and keep the test file larger than any cache in the path. If --direct=1 then fails with err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1, run df -hT .: a Type of overlay does not support O_DIRECT, so point the test at real storage instead.

Is yabs.sh enough on its own?

For a first look, yes. It runs fio at four block sizes, iperf3 in both directions and Geekbench, and it prints one summary other people can read. It stops being enough when you want to know why a number is what it is, because you cannot vary its flags per test. Once a yabs result looks wrong, reproduce it with fio or sysbench directly and change one flag at a time.

Which single number predicts how my application will feel?

Single core CPU speed and 4k random read latency, in that order, for most web and database workloads. Throughput figures look impressive and rarely decide anything, because a typical request is small. Quote the 99th percentile from the fio clat percentiles block rather than the average, since the slow one request in a hundred is the one a user notices.

Do I need to install anything before benchmarking?

fio, sysbench and iperf3 are all in the Ubuntu and Debian archives: sudo apt install -y fio sysbench iperf3. yabs.sh needs only curl, because it downloads static binaries for anything missing. Delete every test file when you finish, since a 2G fio file left behind on a 20G disk becomes somebody's disk full alert weeks later.