How to Benchmark VPS Well: yabs.sh, fio and More

Wetin e mean to benchmark a VPS

To benchmark a VPS, you measure four things: how fast one CPU core runs, how much memory bandwidth the machine get, how many small random disk operations the storage fit serve every second, and how much throughput the network link dey deliver. One run of yabs.sh gives you all four in about ten minutes. To read the result na the harder part, because a VPS (virtual private server) dey share physical hardware with other tenants. So, the same machine fit report one number for 03:00 and another very different one for 20:00.

The plan here na to run yabs.sh for a quick picture, then run the tools wey dey underneath am by hand. When you run dem yourself, you fit change one flag, monitor how the number dey move, and learn wetin that number really dey measure. Do this after you don set up the machine, no be before. The steps for the first ten minutes for a new VPS come first, because a box wey still dey apply e first round of updates go benchmark badly for reasons wey no get anything to do with the hardware.

Check the machine before you measure am

Half of every bad benchmark na machine wey the author no understand.

nproc
lscpu | grep -E 'Model name|Hypervisor|Thread'
free -h
df -hT /
uname -r
systemd-detect-virt

Hypervisor vendor: KVM mean full virtualisation, so you dey run your own kernel. systemd-detect-virt printing lxc or openvz mean container virtualisation instead: you dey share the host kernel, and your CPU and memory limits na cgroup (control group) settings, no be virtual hardware. For cgroup v2 system, you fit read the CPU limit directly.

cat /sys/fs/cgroup/cpu.max

max 100000 mean say quota no dey. 200000 100000 mean say you fit use 200000 microseconds of CPU for every 100000 microsecond period, wey be quota for two cores. Plan wey dem advertise as 4 vCPU but get quota for two cores no go ever score like four cores, and no benchmark tool go print line wey tell you why.

df -hT / matter for another reason: the Type column. If e read overlay, you dey inside container, and the disk test below need change. Note am now.

Monitor steal time all the time

Steal time na the part of time wey your virtual CPU ready to run, but hypervisor give the physical core to another person. Na the most useful single signal to show say the result come from your neighbours, no be from the hardware.

vmstat 1 10

Read the st column for the right side. Steady 0 or 1 normal. Values wey dey stay above 5 mean say the host get more workload than e fit handle for that moment. So every CPU number wey you record for that period go low, and no be your machine cause am. top dey show the same value as %st for the CPU line. Keep vmstat 1 dey run for another SSH session while you dey benchmark, and write the steal value beside each result.

Start with yabs.sh

yabs.sh (Yet Another Bench Script) na shell script wey dey download static fio, iperf3 and Geekbench binaries, run dem, then print one summary. Na the common language for VPS benchmark discussions, so yabs output na the fastest way to compare results with another person.

The project own one-line form be this.

curl -sL yabs.sh | bash

That command pipes anything wey the URL serve today straight into a shell. Download am, read am, then run am.

curl -sLo yabs.sh https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh
less yabs.sh
bash yabs.sh

Put flags after -s -- when you pipe, or directly after the filename when you run local copy. The useful ones be: -f skips the disk test, -i skips the network test, -g skips Geekbench, -r reduces the iperf3 locations to two, -j prints the results as JSON, and -w results.json writes that JSON to a file.

bash yabs.sh -r -w yabs-run1.json

Know these two things before the first run. Geekbench uploads your result and prints public browser.geekbench.com URL, so anybody wey get that link fit read your CPU model and scores. -g skips that test completely. Second, the iperf3 stage sends real network traffic to servers for several regions, and that traffic counts against your monthly bandwidth allowance. For 1 Gbit/s link, full network stage fit move tens of gigabytes, so use -r when your allowance small and -i for metered link.

Wetin each part of the yabs output mean

The disk section dey run fio with 50/50 read and write mix for four block sizes: 4k, 64k, 512k and 1m. E dey report IOPS (input/output operations per second) and bandwidth for each one. Na the 4k row you suppose care about for database, mail server, or anything wey dey do plenty small writes, because most server IO dey small and scattered. Na the 1m row be the one for backups and video, where you dey move long runs of bytes.

The network section dey run iperf3 against public servers for different regions, for both directions, using parallel streams. Treat low number here as question, no be answer, because public iperf3 servers dey shared and dem often dey saturated, so poor result fit come from the far end.

The Geekbench section dey give one core score and multi core score. Single core dey predict how fast one request, one compile, or one query go finish. Multi core mostly dey tell you how many cores you really get.

Disk: run fio yourself

fio (flexible IO tester) na di tool wey dey under the yabs disk section. Na when you run am directly you go start understand wetin the flags mean.

sudo apt update && sudo apt install -y fio sysbench iperf3

Na 4k random read test for queue depth 32, on the filesystem wey you really care about:

fio --name=randread4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randread --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting

The summary line wey you suppose read from the output look like this.

read: IOPS=184k, BW=719MiB/s (754MB/s)(42.1GiB/60001msec)

Under am, fio dey print a clat percentiles block. The 99.00th percentile na the figure wey make sense to quote, because e show how long the slowest one request out of every hundred wait. Average latency dey hide exactly the stalls wey user go notice.

• --direct=1 dey open the file with O_DIRECT, so reads bypass the kernel page cache. Without am, the second pass over a 2G file for machine wey get 8G RAM go come from memory, and fio go report IOPS for millions. That number real, but na memory number.
• --ioengine=libaio dey submit asynchronous requests, and na wetin let --iodepth=32 keep 32 of dem in flight. With synchronous engine like psync, iodepth wey pass 1 no dey do anything at all, so you dey measure one request at a time.
• --time_based --runtime=60 dey run for fixed 60 seconds instead of fixed amount of work. This one make fast disk and slow disk get the same wall clock, so comparison remain fair.
• --size=2G dey set the test file size. Keep am bigger than any cache for the path, and first check say free space dey available.

Random write na the same command with --rw=randwrite. Run am separately, then delete the file.

fio --name=randwrite4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randwrite --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting
rm -f ./fio-testfile

For mix wey near real traffic more, use --rw=randrw --rwmixread=70. The class of storage wey you dey use changes these results more than any flag. That difference dey covered for the difference between NVMe and SATA SSD storage on a VPS.

When fio stop with Unknown error -1

Direct IO no dey available for every filesystem. overlay, the filesystem wey Docker dey give container by default, and some network filesystems no support O_DIRECT. So, libaio dey submit request wey kernel no fit complete, and fio dey stop:

fio: io_u error on file ./fio-testfile: Unknown error -1: read offset=0, buflen=4096
fio: pid=1234, err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1

Run df -hT . first. If the Type column talk say overlay, point --filename to path for real storage, like bind mounted volume, or run fio for host instead of inside container. If real storage no dey available, buffered synchronous run still fit prove say the command itself correct.

fio --name=randread4k-buffered --filename=./fio-testfile --size=256M --bs=4k \
  --rw=randread --ioengine=psync --direct=0 --numjobs=1 \
  --runtime=15 --time_based --group_reporting
rm -f ./fio-testfile

Talk true about wetin that run dey measure. After the first pass, the 256M file dey inside page cache, so the IOPS figure dey describe your RAM. Use am confirm say fio install correctly and the flags parse. No ever present am as disk result.

Why dd no be disk benchmark

dd dey show for plenty VPS discussions, and e dey answer one narrow question.

dd if=/dev/zero of=./ddtest bs=1M count=1024 oflag=direct conv=fdatasync
rm -f ./ddtest

E dey measure sequential write throughput with one thread and one request in flight. E fit work as basic sanity check. E no talk anything about random IO, and e no talk wetin go happen when 32 requests reach at once. If you remove oflag=direct, e mostly measure how fast your kernel accept writes into memory. Na why dd figures wey people quote for forum posts dey often absurd.

CPU: sysbench cpu

sysbench cpu --cpu-max-prime=20000 --threads=1 run
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

The figure wey matter na events per second. Run am with one thread first. Na this number dey show how fast one PHP request go finish or one compile job go complete. E dey vary pass among hosts wey get the same price. Then run am with every thread. This one go show whether your vCPUs na separate cores or slices from one core.

Make you understand wetin this test dey measure: sysbench cpu dey repeatedly find prime numbers with 64 bit integer arithmetic. E no dey stress memory bandwidth, vector units, or cache in any way wey resemble real workload. So e good to rank two hosts, but e no good to predict how your application go run.

Ubuntu 24.04 ships sysbench 1.0.20, and for this version the test name dey come first. If you copy command wey use --test=cpu from old post, you go get WARNING: the --test option is deprecated. Scores from sysbench 0.4 and sysbench 1.0 no comparable at all. So never compare your result with published number wey no state the version.

Memory: sysbench memory

sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=write --threads=1 run
sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=read --threads=1 run

Result dey come out for MiB/sec, and reads dey faster pass writes for every machine. Keep --memory-block-size for 1M, and make e remain the same for every host wey you compare. For 1K, the number dey drop sharply because you dey pay per-operation overhead one thousand times more often. So, na loop cost you dey measure instead of memory bandwidth. This na the flag wey people mismatch pass for published memory scores.

Network: iperf3

The correct way to test throughput na against a second machine wey you control, because then you know wetin both ends dey do.

For the far end:

iperf3 -s

That one dey listen for TCP 5201. Open the port only for the address wey you dey test from, and close am when you finish. Basic ufw firewall rules for VPS explain the syntax.

From the VPS wey you dey test:

iperf3 -c 203.0.113.10 -t 30
iperf3 -c 203.0.113.10 -t 30 -R
iperf3 -c 203.0.113.10 -t 30 -P 8

The first one dey measure upload from the machine wey you dey test. -R reverse the direction, so e dey measure download. -P 8 open eight parallel streams.

Run both the single stream and the parallel version, because dem answer different questions. One TCP connection fit hold only the amount of unacknowledged data wey the window allow, so the limit na roughly the window size divided by the round trip time. With 80 ms latency and 4 MB window, that limit na about 400 Mbit/s, no matter how fast the link underneath be. The single stream figure tell you wetin one download go get. The parallel figure tell you the link capacity.

Monitor your bandwidth allowance while you dey do this. Thirty seconds at 1 Gbit/s dey move about 3.75 GB, and you go run am several times for each direction.

Reference figures, and how to read yours

[
  {
    "device": "Local NVMe",
    "iops_4k_read": "180,000"
  },
  {
    "device": "Local SATA SSD",
    "iops_4k_read": "90,000"
  },
  {
    "device": "Network block",
    "iops_4k_read": "12,000"
  },
  {
    "device": "Spinning disk",
    "iops_4k_read": "180"
  }
]

A local NVMe volume for published results normally dey reach around 180,000 4k random read IOPS. Local SATA SSD dey around 90,000. Network attached block storage, where every request dey cross network before e reach disk, dey closer to 12,000, while spinning disk dey manage about 180, because e dey move physical head for every random request.

These figures na the usual published results for each storage class, no be measurements from one host. Use dem for one thing: to check say your own result dey within the correct order of magnitude. If plan wey dem sell as NVMe dey benchmark for low thousands of 4k IOPS, first confirm say --direct=1 dey on. If e dey on, then either the storage no be wetin the product page describe, or you dey share am with neighbour wey dey very busy.

Why one run no be benchmark

One result na snapshot of one minute for a shared machine. Treat am as one sample.

• Run every test at least five times, spread across different hours and at least two different days. Keep the median and the spread. Result wey dem publish without a spread na marketing figure.
• Record the steal time beside each run. Throw out runs where st high, or at minimum note say e high.
• Run the disk test at two durations. Many plans dey give burst IOPS allowance wey dey refill over time, so a 60 second fio run dey measure the burst while --runtime=600 dey measure the floor. The floor na wetin you go get for bad day.
• Check say nothing else dey run. unattended-upgrades starting an apt transaction in the middle of a CPU test dey cost you real points, and ps -e -o comm= | grep -E 'apt|dpkg' before each run dey take one second.
• Change one variable at a time. Different tool versions, block sizes or thread counts dey produce numbers wey you no fit compare, no matter how similar dem look.

When you compare two providers, run dem for the same hour of the same day. Otherwise, you don measure the time of day.

Benchmark your own workload last

Synthetic tools dey rank machines. Na only your own workload fit tell you whether a machine enough. Time the thing wey you really dey do.

time tar -czf /tmp/bench.tgz /usr/share
rm -f /tmp/bench.tgz

That one dey compress some hundred megabytes, so e dey test CPU and disk together, and e dey change whenever either one change. The Removing leading / from member names warning normal. Better still, time your own build, your own slowest query, or your own page render. If build dey take 4 minutes for one host and 7 for another, the answer don clear, no matter wetin Geekbench think. This na also the measurement wey go tell you when getting more machine no longer worth the money, and e good make you know this before you read wetin VPS dey really cost per month or move the workload go dedicated server.

FAQ

Why I dey get different benchmark result every time I run am?

A VPS dey share physical CPU, storage and network with other tenants, so wetin dem dey do for that moment fit affect your result. Run vmstat 1 during the test and read the st column: if steal time stay above 5, e mean say the host dey busy, and your CPU score low for reasons wey no come from your machine. The correct answer na better method, no be tuning. Run each test five or more times across different hours, then report the median together with the spread.

Why fio dey report millions of IOPS?

Almost always na because --direct=1 dey miss. Without am, fio dey read through the kernel page cache. So after the first pass, RAM dey serve the 2G test file, and wetin you measure na memory bandwidth. Add --direct=1 and make the test file bigger than any cache for the path. If --direct=1 fail with err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1, run df -hT .: a Type of overlay no support O_DIRECT, so point the test to real storage instead.

yabs.sh alone enough?

For first look, yes. E dey run fio with four block sizes, iperf3 for both directions and Geekbench, and e dey print one summary wey other people fit read. E no dey enough again when you wan know why one number be wetin e be, because you no fit change the flags for each test. Once yabs result look wrong, reproduce am directly with fio or sysbench and change one flag at a time.

Which single number dey predict how my application go feel?

Single core CPU speed and 4k random read latency, for that order, na the important ones for most web and database workloads. Throughput figures dey look impressive but dem rarely decide anything, because typical request small. Quote the 99th percentile from the fio clat percentiles block instead of the average, because na the one slow request out of 100 wey user go notice.

I need install anything before benchmarking?

fio, sysbench and iperf3 dey inside the Ubuntu and Debian archives: sudo apt install -y fio sysbench iperf3. yabs.sh only need curl, because e dey download static binaries for anything wey dey miss. Delete every test file when you finish, because if you leave 2G fio file for 20G disk, e fit become somebody disk full alert weeks later.