How to Benchmark VPS Well Without Fooling Yourself

Wetin benchmarking VPS mean

To benchmark VPS na to measure four things: how fast one CPU core dey run, how much memory bandwidth the machine get, how many small random disk operations the storage fit serve every second, and how much throughput the network link dey deliver. One run of yabs.sh go give you all four for about ten minutes. To understand the result na the harder part, because VPS (virtual private server) dey share physical hardware with other tenants. So the same machine fit report one number for 03:00 and another number wey differ well-well for 20:00.

The plan na to run yabs.sh for quick overview, then run the tools wey dey underneath am by hand. When you run dem yourself, you fit change one flag, watch how the number move, and learn wetin that number really dey measure. Do this after you don set up the machine, no be before. The steps for the first ten minutes on a new VPS come first, because box wey still dey apply its first round of updates go benchmark badly for reasons wey no concern the hardware.

Check the machine before you measure am

Half of every bad benchmark na machine wey the author no understand.

nproc
lscpu | grep -E 'Model name|Hypervisor|Thread'
free -h
df -hT /
uname -r
systemd-detect-virt

Hypervisor vendor: KVM mean full virtualisation, so you dey run your own kernel. systemd-detect-virt wey dey print lxc or openvz mean container virtualisation instead: you dey share the host kernel, and your CPU and memory limits na cgroup (control group) settings, no be virtual hardware. For cgroup v2 system, you fit read the CPU limit directly.

cat /sys/fs/cgroup/cpu.max

max 100000 mean say quota no dey. 200000 100000 mean say you fit use 200000 microseconds of CPU for every 100000 microsecond period, wey be quota equal to two cores. Plan wey dem advertise as 4 vCPU but get quota of two cores no go ever score like four cores, and no benchmark tool go print line wey explain why.

df -hT / matter for another reason: the Type column. If e read overlay, you dey inside container, and the disk test below need change. Note am now.

Monitor steal time the whole time

Steal time na the portion of time wey your virtual CPU ready to run but the hypervisor give the physical core to another person. Na the most useful single signal to tell say result concern your neighbours, no be the hardware.

vmstat 1 10

Read the st column for the right side. Steady 0 or 1 normal. Values wey stay above 5 mean say host get too many workloads for that moment, so every CPU number wey you record for that period low, and no be your machine cause am. top show the same figure as %st for the CPU line. Keep vmstat 1 running for another SSH session while you dey benchmark, then write the steal figure beside each result.

Start with yabs.sh

yabs.sh (Yet Another Bench Script) na shell script wey dey download static fio, iperf3 and Geekbench binaries, run dem, then print one summary. Na the common language for VPS benchmark discussions, so yabs output na the fastest way to compare result with another person.

The project own one-line form na this.

curl -sL yabs.sh | bash

That command pipe anything wey the URL serve today straight go shell. Download am, read am, then run am.

curl -sLo yabs.sh https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh
less yabs.sh
bash yabs.sh

Put flags after -s -- when you pipe, or put dem straight after filename when you run local copy. The useful ones be: -f skip disk test, -i skip network test, -g skip Geekbench, -r reduce iperf3 locations to two, -j print result as JSON, and -w results.json write that JSON go file.

bash yabs.sh -r -w yabs-run1.json

Know two things before the first run. Geekbench uploads your result and prints public browser.geekbench.com URL, so anybody wey get that link fit read your CPU model and scores. -g skip that test completely. Second, the iperf3 stage dey move real traffic go servers for several regions, and that one count against your monthly bandwidth allowance. For 1 Gbit/s link, complete network stage fit move tens of gigabytes, so use -r when your allowance small and -i for metered link.

Wetin each part of the yabs output mean

The disk section dey run fio with 50/50 read and write mix for four block sizes: 4k, 64k, 512k and 1m. E dey report IOPS (input/output operations per second) and bandwidth for each one. Na the 4k row you suppose focus on for database, mail server, or anything wey dey do plenty small writes, because most server IO small and scattered. Na the 1m row fit backups and video, where you dey move long runs of bytes.

The network section dey run iperf3 against public servers for different regions, for both directions, using parallel streams. Treat low number for here as question, no be final answer, because dem dey share public iperf3 servers and dem often dey saturated. So poor result fit come from the far end.

The Geekbench section dey give single core score and multi core score. Single core dey predict how fast one request, one compile, or one query go finish. Multi core mostly dey show how many cores you really get.

Disk: run fio by yourself

fio (flexible IO tester) na di tool wey dey underneath the yabs disk section. Na when you run am directly be the time the flags start to get clear meaning.

sudo apt update && sudo apt install -y fio sysbench iperf3

For 4k random read test with queue depth 32, for the filesystem wey you really care about:

fio --name=randread4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randread --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting

The summary line wey you suppose read from the output look like this.

read: IOPS=184k, BW=719MiB/s (754MB/s)(42.1GiB/60001msec)

Under am, fio dey print one clat percentiles block. The 99.00th percentile na the figure wey worth quoting, because e show how long the slowest one request out of every hundred wait. Average latency fit hide exactly the stalls wey user go notice.

• --direct=1 dey open the file with O_DIRECT, so reads bypass kernel page cache. Without am, the second pass over 2G file for machine wey get 8G RAM go come from memory, and fio go report IOPS for millions. That number real, but na memory number.
• --ioengine=libaio dey submit asynchronous requests, and na wetin let --iodepth=32 keep 32 requests in flight. With synchronous engine like psync, iodepth above 1 no dey do anything, so you dey measure one request at a time.
• --time_based --runtime=60 dey run for fixed 60 seconds instead of fixed amount of work. This make fast disk and slow disk use the same wall clock time, so the comparison remain fair.
• --size=2G dey set the test file size. Make am bigger than any cache for the path, and first check say free space dey enough.

Random write na the same command with --rw=randwrite. Run am separately, then delete the file.

fio --name=randwrite4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randwrite --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting
rm -f ./fio-testfile

For mix wey dey closer to real traffic, use --rw=randrw --rwmixread=70. The class of storage wey you dey use affects these results more than any flag, and the difference between NVMe and SATA SSD storage on a VPS cover that split.

When fio stop with Unknown error -1

Direct IO no dey available for every filesystem. overlay, wey na the filesystem Docker dey give container by default, and some network filesystems no support O_DIRECT. So libaio dey submit request wey kernel no fit complete, and fio go stop:

fio: io_u error on file ./fio-testfile: Unknown error -1: read offset=0, buflen=4096
fio: pid=1234, err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1

Run df -hT . first. If the Type column show overlay, point --filename to path for real storage, like bind mounted volume, or run fio for the host instead of inside the container. If real storage no dey accessible, buffered synchronous run still fit prove say the command itself correct.

fio --name=randread4k-buffered --filename=./fio-testfile --size=256M --bs=4k \
  --rw=randread --ioengine=psync --direct=0 --numjobs=1 \
  --runtime=15 --time_based --group_reporting
rm -f ./fio-testfile

Make you understand wetin that run mean. After the first pass, the 256M file dey inside page cache, so the IOPS figure dey describe your RAM. Use am confirm say fio install correctly and the flags parse. Never quote am as disk result.

Why dd no be disk benchmark

dd dey appear for plenty VPS threads, and e dey answer one narrow question.

dd if=/dev/zero of=./ddtest bs=1M count=1024 oflag=direct conv=fdatasync
rm -f ./ddtest

This one dey measure sequential write throughput with one thread and one request for queue. E be reasonable sanity check. E no tell you anything about random IO, and e no tell you wetin go happen when 32 requests land at once. Remove oflag=direct and e mostly go measure how fast your kernel accept writes into memory. Na why dd figures wey people quote for forum posts often dey absurd.

CPU: sysbench cpu

sysbench cpu --cpu-max-prime=20000 --threads=1 run
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

The number wey matter na events per second. Run am single-threaded first. Na this number dey decide how quickly one PHP request go finish or one compile job go complete, and na here hosts wey cost the same dey differ pass. Then run am with every thread. This one go show whether your vCPUs na separate cores or slices from one core.

Make you clear about wetin this dey measure: sysbench cpu dey repeatedly find prime numbers with 64 bit integer arithmetic. E no dey stress memory bandwidth, vector units, or cache for any way wey resemble real workload. So e good for ranking two hosts, but e no good for predicting how your application go run.

Ubuntu 24.04 ships sysbench 1.0.20, where the test name dey come first. If you copy command wey get --test=cpu from old post, you go get WARNING: the --test option is deprecated. Scores from sysbench 0.4 and sysbench 1.0 no comparable at all, so never compare your result with published number wey no state the version.

Memory: sysbench memory

sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=write --threads=1 run
sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=read --threads=1 run

Result dey for MiB/sec, and read dey faster pass write for every machine. Keep --memory-block-size for 1M, and make sure e dey the same for every host wey you compare. For 1K the number dey collapse, because you dey pay per-operation overhead one thousand times more often. So, wetin you dey measure na loop cost instead of memory bandwidth. Na this flag dem dey mismatch pass for published memory scores.

Network: iperf3

The correct way to test throughput na against a second machine wey you control, because then you know wetin both ends dey do.

For the far end:

iperf3 -s

That one dey listen on TCP 5201. Open the port only for the address wey you dey test from, and close am when you finish. Basic ufw firewall rules for VPS cover the syntax.

From the VPS wey you dey test:

iperf3 -c 203.0.113.10 -t 30
iperf3 -c 203.0.113.10 -t 30 -R
iperf3 -c 203.0.113.10 -t 30 -P 8

The first one dey measure upload from the machine wey you dey test. -R reverse the direction, so e measure download. -P 8 open eight parallel streams.

Run both the single stream and the parallel version, because dem dey answer different questions. One TCP connection fit only carry the amount of unacknowledged data wey e window allow, so the limit na roughly window size divided by round trip time. For 80 ms latency with 4 MB window, that limit na about 400 Mbit/s, no matter how fast the link underneath be. The single stream result tell you wetin one download go get. The parallel result tell you the capacity of the link.

Monitor your bandwidth allowance while you dey do this. Thirty seconds for 1 Gbit/s go move about 3.75 GB, and you go run am several times for each direction.

Reference figures, and how to read yours

[
  {
    "device": "Local NVMe",
    "iops_4k_read": "180,000"
  },
  {
    "device": "Local SATA SSD",
    "iops_4k_read": "90,000"
  },
  {
    "device": "Network block",
    "iops_4k_read": "12,000"
  },
  {
    "device": "Spinning disk",
    "iops_4k_read": "180"
  }
]

Local NVMe volume wey dey show for published results usually dey near 180,000 4k random read IOPS. Local SATA SSD dey around 90,000. Network attached block storage, where every request dey cross network before e reach disk, dey nearer 12,000, while spinning disk dey manage roughly 180, because e dey move physical head for every random request.

These figures na normal published results for each storage class; dem no be measurement from one host. Use dem for one thing: to check whether your own result dey within the correct order of magnitude. If plan wey dem sell as NVMe benchmark for low thousands of 4k IOPS, first confirm say --direct=1 dey on. If e dey on, then either the storage no be wetin the product page describe, or you dey share am with neighbour wey dey very busy.

Why one run no be benchmark

One result na snapshot of one minute for a shared machine. Treat am as one sample.

• Run every test at least five times, spread am across different hours and at least two different days. Keep the median and the spread. Result wey dem publish without spread na marketing figure.
• Record the steal time beside each run. Throw away runs where st high, or at least note say e high.
• Run the disk test for two durations. Many plans get burst IOPS allowance wey dey refill over time, so 60 second fio run dey measure the burst while --runtime=600 dey measure the floor. The floor na wetin you go get for bad day.
• Check say nothing else dey run. unattended-upgrades starting an apt transaction in the middle of CPU test go cost you real points, and ps -e -o comm= | grep -E 'apt|dpkg' before each run dey take one second.
• Change one variable at a time. Different tool versions, block sizes, or thread counts dey produce numbers wey you no fit compare, no matter how similar dem look.

When you compare two providers, run dem for the same hour of the same day. Otherwise, na time of day you don measure.

Benchmark your own workload last

Synthetic tools dey rank machines. Na only your own workload fit tell you whether machine don sufficient. Time the actual thing wey you dey do.

time tar -czf /tmp/bench.tgz /usr/share
rm -f /tmp/bench.tgz

That one dey compress a few hundred megabytes, so e dey test CPU and disk together, and result go change whenever either one change. The Removing leading / from member names warning dey normal. Better still, time your own build, your own slowest query, or your own page render. If build dey take 4 minutes for one host and 7 for another, the matter don clear, no matter wetin Geekbench talk. This na also the measurement wey go tell you when extra machine no longer worth the money, and e good make you know this before you read wetin VPS dey cost per month or move the workload go dedicated server.

FAQ

Why I dey get different benchmark result every time I run am?

A VPS dey share physical CPU, storage and network with other tenants, so wetin dem dey do for that moment go affect your result. Run vmstat 1 during the test and read the st column: if sustained steal time pass 5, e mean say host busy, and your CPU score low because of something outside your machine. The correct answer na better method, no be tuning. Run each test five times or more across different hours, then report the median together with the spread.

Why fio dey report millions of IOPS?

Almost every time, na because --direct=1 no dey. Without am, fio dey read through kernel page cache, so after the first pass, RAM dey serve the 2G test file and wetin you measure na memory bandwidth. Add --direct=1 and make the test file bigger than any cache for the path. If --direct=1 fail with err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1, run df -hT .: a Type of overlay no support O_DIRECT, so point the test to real storage instead.

Yabs.sh enough by itself?

For first look, yes. E dey run fio with four block sizes, iperf3 for both directions, and Geekbench, then e print one summary wey other people fit read. E no remain enough when you want know why one number be as e be, because you no fit change the flags for each test. Once yabs result look wrong, reproduce am directly with fio or sysbench, and change one flag at a time.

Which single number dey predict how my application go feel?

Single-core CPU speed and 4k random read latency, in that order, for most web and database workloads. Throughput figures fit look impressive but dem rarely decide anything, because typical request small. Quote the 99th percentile from the fio clat percentiles block instead of the average, because na the one slow request out of 100 wey user go notice.

I need install anything before benchmarking?

fio, sysbench and iperf3 dey inside Ubuntu and Debian archives: sudo apt install -y fio sysbench iperf3. yabs.sh only need curl, because e dey download static binaries for anything wey no dey. Delete every test file when you finish, because if you leave 2G fio file for 20G disk, e fit become somebody disk-full alert weeks later.