How to Benchmark VPS Well Without Fooling Yourself

Wetin benchmarking a VPS mean

To benchmark a VPS, you dey measure four things: how fast one CPU core dey run, how much memory bandwidth the machine get, how many small random disk operations the storage fit serve every second, and how much throughput the network link dey deliver. One run of yabs.sh go give you CPU, disk, and network results for about ten minutes; use the separate memory test below for memory bandwidth. The hard part na to read the result, because a VPS (virtual private server) dey share physical hardware with other tenants. So, the same machine fit report one number for 03:00 and another very different number for 20:00.

The plan na to run yabs.sh for quick overview, then run the tools underneath am by hand. When you run dem yourself, you fit change one flag, monitor how the number dey move, and understand wetin that number really dey measure. Do this after you don set up the machine, not before. The steps for the first ten minutes for a new VPS come first, because a box wey still dey apply im first round of updates go benchmark badly for reasons wey no relate to the hardware.

Check the machine before you measure am

Half of every bad benchmark come from machine wey the author no understand.

nproc
lscpu | grep -E 'Model name|Hypervisor|Thread'
free -h
df -hT /
uname -r
systemd-detect-virt

Hypervisor vendor: KVM mean full virtualisation, so you dey run your own kernel. If systemd-detect-virt dey print lxc or openvz, e mean container virtualisation instead: you share the host kernel, and your CPU and memory limits na cgroup (control group) settings, no be virtual hardware. For cgroup v2 system, you fit read the CPU limit from your container cgroup; the path wey follow apply when dem mount that cgroup for /sys/fs/cgroup.

cat /sys/fs/cgroup/cpu.max

max 100000 mean this cgroup no get quota of im own; ancestor cgroups still fit impose one. 200000 100000 mean you fit use 200000 microseconds of CPU for every 100000 microsecond period, wey be quota for two cores. Plan wey dem advertise as 4 vCPU with quota for two cores no go ever score like four cores. Benchmark tools fit identify that restriction: VoxelBench dey report CPU quotas and flag shared or throttled CPU when host expose much more CPUs pass wetin the server fit use.

df -hT / matter for another reason: the Type column. If e read overlay, you dey inside container, and your disk results go describe any storage wey dey underneath that overlay. Note am now.

Monitor steal time throughout

Steal time na the part of time wey your virtual CPU spend dey wait without choosing, while hypervisor no schedule am. E dey show say CPU time no dey available, but e no fit tell whether busy neighbours cause am or provider dey enforce your CPU quota.

vmstat 1 10

Read the st column for the right side. If steal time dey continue, e mean CPU time no dey available to the guest, but value wey pass 5% no prove say host get too many workloads. Providers fit deliberately show CPU quota enforcement as steal time. top dey show the same figure as %st for the CPU line. Keep vmstat 1 running for another SSH session while you dey benchmark, then write the steal figure beside each result.

Start with yabs.sh

yabs.sh (Yet Another Bench Script) na shell script wey dey run fio, iperf3 and Geekbench, then print one summary. E go use fio and iperf3 if dem don already install, and e go download static binaries if dem no dey. Na the common language for VPS benchmark discussions, so yabs output na the fastest way to compare result with another person.

The project own one-line form na this.

curl -sL yabs.sh | bash

That command pipe anything wey the URL serve today straight into shell. Download am, read am, then run am.

curl -sLo yabs.sh https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/yabs.sh
less yabs.sh
bash yabs.sh

Put flags after -s -- when you pipe am, or directly after filename when you run local copy. The useful ones be these: -f skip disk test, -i skip network test, -g skip Geekbench, -r reduce iperf3 locations to three, -j print result as JSON, and -w results.json write that JSON to a file.

bash yabs.sh -r -w yabs-run1.json

You need know two things before the first run. Geekbench uploads your result and prints public browser.geekbench.com URL, so anybody wey get that link fit read your CPU model and scores. -g skip that test completely. Second, the iperf3 stage dey send real traffic to servers for different regions, and that traffic go count against your monthly bandwidth allowance. For 1 Gbit/s link, full network stage fit move tens of gigabytes, so use -r when your allowance small and -i for metered link.

Wetin each part of the yabs output mean

The disk section dey run fio with 50/50 read and write mix for four block sizes: 4k, 64k, 512k and 1m. E dey report IOPS (input/output operations per second) and bandwidth for each one. Na the 4k row you suppose focus on for database, mail server, or anything wey dey do plenty small writes, because most server IO small and scattered. Na the 1m row fit backups and video, where you dey move long runs of bytes.

The network section dey run iperf3 against public servers for different regions, for both directions, using parallel streams. Treat low number for here as question, no be final answer, because dem dey share public iperf3 servers and dem often dey saturated. So poor result fit come from the far end.

The Geekbench section dey give single core score and multi core score. Single core dey predict how fast one request, one compile, or one query go finish. Multi core mostly dey show how many cores you really get.

Disk: run fio by yourself

fio (flexible IO tester) na di tool wey dey underneath the yabs disk section. Na when you run am directly be the time the flags start to get clear meaning.

sudo apt update && sudo apt install -y fio sysbench iperf3

For 4k random read test with queue depth 32, for the filesystem wey you really care about:

fio --name=randread4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randread --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting

The summary line wey you suppose read from the output look like this.

read: IOPS=184k, BW=719MiB/s (754MB/s)(42.1GiB/60001msec)

Under am, fio dey print one clat percentiles block. The 99.00th percentile na the figure wey worth quoting, because e show how long the slowest one request out of every hundred wait. Average latency fit hide exactly the stalls wey user go notice.

• --direct=1 dey open the file with O_DIRECT, so reads bypass kernel page cache. Without am, the second pass over 2G file for machine wey get 8G RAM go come from memory, and fio go report IOPS for millions. That number real, but na memory number.
• --ioengine=libaio dey submit asynchronous requests, and na wetin let --iodepth=32 keep 32 requests in flight. With synchronous engine like psync, iodepth above 1 no dey do anything, so you dey measure one request at a time.
• --time_based --runtime=60 dey run for fixed 60 seconds instead of fixed amount of work. This make fast disk and slow disk use the same wall clock time, so the comparison remain fair.
• --size=2G dey set the test file size. Make am bigger than any cache for the path, and first check say free space dey enough.

Random write na the same command with --rw=randwrite. Run am separately, then delete the file.

fio --name=randwrite4k --filename=./fio-testfile --size=2G --bs=4k \
  --rw=randwrite --ioengine=libaio --iodepth=32 --direct=1 \
  --runtime=60 --time_based --group_reporting
rm -f ./fio-testfile

For mix wey dey closer to real traffic, use --rw=randrw --rwmixread=70. The class of storage wey you dey use affects these results more than any flag, and the difference between NVMe and SATA SSD storage on a VPS cover that split.

When fio stop with Unknown error -1

Direct IO no dey available for every filesystem. So when fio fail for im first request with --direct=1 set, direct IO na the first thing to rule out. The failure fit look like this:

fio: io_u error on file ./fio-testfile: Unknown error -1: read offset=0, buflen=4096
fio: pid=1234, err=-1/file:ioengines.c:321, func=get_events, error=Unknown error -1

Run df -hT . first. A Type of overlay, wey na the filesystem Docker dey give container by default, no be the cause by itself. overlayfs dey pass O_DIRECT through to the filesystem underneath, and e go reject am only when that filesystem no fit do direct IO. So --direct=1 --ioengine=libaio go work for overlay wey dey sit on ext4 or xfs. When the filesystem under the test file truly no fit do direct IO, fio usually go talk am as e dey open the file: fio: looks like your file system does not support direct=1/buffered=0. For that case, point --filename to a path for real local storage, like a bind mounted volume, or run fio for the host instead of inside the container. If real storage no dey reachable, buffered synchronous run still fit prove say the command itself correct.

fio --name=randread4k-buffered --filename=./fio-testfile --size=256M --bs=4k \
  --rw=randread --ioengine=psync --direct=0 --numjobs=1 \
  --runtime=15 --time_based --group_reporting
rm -f ./fio-testfile

Be honest about wetin that run mean. After the first pass, the 256M file dey inside page cache, so the IOPS figure dey describe your RAM. Use am confirm say fio install and the flags parse correctly. Never quote am as disk result.

Why dd no be disk benchmark

dd dey appear for plenty VPS threads, and e dey answer one narrow question.

dd if=/dev/zero of=./ddtest bs=1M count=1024 oflag=direct conv=fdatasync
rm -f ./ddtest

E dey measure sequential write throughput with one thread and one request in flight. E be reasonable sanity check. E no talk anything about random IO, and e no talk wetin go happen when 32 requests enter at once. Remove oflag=direct and writes go pass through page cache, but conv=fdatasync still force the output data go storage before command finish. Remove that one too, and the result fit mostly measure how quickly your kernel accept writes into memory. Na why dd figures wey people quote for forum posts dey often absurd.

CPU: sysbench cpu

sysbench cpu --cpu-max-prime=20000 --threads=1 run
sysbench cpu --cpu-max-prime=20000 --threads=$(nproc) run

The number wey you need keep na events per second. Run am with one thread first. Na this number dey decide how fast one PHP request go finish or one compile job go complete, and na here hosts wey cost the same dey differ pass. Then run am with all threads. This one show how throughput dey scale across the vCPUs wey dem allocate give you.

Make you understand wetin this dey measure: sysbench cpu dey repeatedly find prime numbers through trial division. For every candidate, e dey calculate double precision square root, then test divisors with 64 bit integer modulo, comparing every divisor with that floating point limit. So e dey combine integer and floating point work. E no dey stress memory bandwidth, vector units, or cache in any way wey resemble real workload. So e good for ranking two hosts, but e poor for predicting how your application go run.

Ubuntu 24.04 ships sysbench 1.0.20, where the test name dey come first. Copy command wey get --test=cpu from old post, and you go get WARNING: the --test option is deprecated. Scores from sysbench 0.4 and sysbench 1.0 no comparable at all. So never compare your result with published number wey no state its version.

Memory: sysbench memory

sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=write --threads=1 run
sysbench memory --memory-block-size=1M --memory-total-size=20G --memory-oper=read --threads=1 run

The result dey for MiB/sec. Keep --memory-block-size for 1M, and make sure say e identical for every host wey you compare. For 1K, the number dey collapse because you dey pay per-operation overhead one thousand times more often. So, wetin you end up measuring na loop cost instead of memory bandwidth. Na this flag people dey mismatch pass for published memory scores.

Network: iperf3

The correct way to test throughput na against a second machine wey you control, because then you know wetin both ends dey do.

For the far end:

iperf3 -s

That one dey listen on TCP 5201. Open the port only for the address wey you dey test from, and close am when you finish. Basic ufw firewall rules for VPS cover the syntax.

From the VPS wey you dey test:

iperf3 -c 203.0.113.10 -t 30
iperf3 -c 203.0.113.10 -t 30 -R
iperf3 -c 203.0.113.10 -t 30 -P 8

The first one dey measure upload from the machine wey you dey test. -R reverse the direction, so e measure download. -P 8 open eight parallel streams.

Run both the single stream and the parallel version, because dem dey answer different questions. One TCP connection fit only carry the amount of unacknowledged data wey e window allow, so the limit na roughly window size divided by round trip time. For 80 ms latency with 4 MB window, that limit na about 400 Mbit/s, no matter how fast the link underneath be. The single stream result tell you wetin one download go get. The parallel result tell you the capacity of the link.

Monitor your bandwidth allowance while you dey do this. Thirty seconds for 1 Gbit/s go move about 3.75 GB, and you go run am several times for each direction.

Reference figures, and how to read yours

[
  {
    "device": "Local NVMe",
    "iops_4k_read": "180,000"
  },
  {
    "device": "Local SATA SSD",
    "iops_4k_read": "90,000"
  },
  {
    "device": "Network block",
    "iops_4k_read": "12,000"
  },
  {
    "device": "Spinning disk",
    "iops_4k_read": "180"
  }
]

Local NVMe volume for published results normally dey near 180,000 4k random read IOPS. Local SATA SSD dey around 90,000. Network attached block storage, where every request dey cross network before e reach disk, dey closer to 12,000. Spinning disk dey manage roughly 180, because e dey move physical head for every random request.

These figures na common published results for each storage class. Dem no be measurements from one host. Use dem for one purpose: check whether your own result dey within the correct order of magnitude. If plan wey dem sell as NVMe dey benchmark for low thousands of 4k IOPS, first confirm say --direct=1 dey on. If e dey on, check the plan's IOPS limits and burst credits, plus storage contention and the advertised hardware.

Why one run no be benchmark

One result na snapshot of one minute for a shared machine. Treat am as one sample.

• Run every test at least five times, spread across different hours and at least two different days. Keep the median and the spread. Result wey you publish without spread na marketing figure.
• Record the steal time beside each run. Keep runs where st high and investigate whether dem show contention or the plan's CPU quota; if you discard dem, you fit hide the performance wey you actually receive.
• Run the disk test for two durations. Many plans get burst IOPS allowance wey dey refill over time, but neither a 60 second fio run nor --runtime=600 guarantee say you measure the floor. AWS gp2 credits fit sustain 3,000 IOPS for at least 30 minutes from full balance, so both runs fit measure burst performance. Check the provider's credit model and monitor the credit balance before you call result sustained performance.
• Check say nothing else dey run. unattended-upgrades starting an apt transaction in the middle of a CPU test go cost you real points, and ps -e -o comm= | grep -E 'apt|dpkg' before each run dey take one second.
• Change one variable at a time. Different tool versions, block sizes or thread counts go produce numbers wey you no fit compare, no matter how similar dem look.

When you compare two providers, run dem for the same hour of the same day. Otherwise, na the time of day you don measure.

Benchmark your own workload last

Synthetic tools dey rank machines. Na only your own workload fit tell you whether machine don sufficient. Time the actual thing wey you dey do.

time tar -czf /tmp/bench.tgz /usr/share
rm -f /tmp/bench.tgz

That one dey compress a few hundred megabytes, so e dey test CPU and disk together, and result go change whenever either one change. The Removing leading / from member names warning dey normal. Better still, time your own build, your own slowest query, or your own page render. If build dey take 4 minutes for one host and 7 for another, the matter don clear, no matter wetin Geekbench talk. This na also the measurement wey go tell you when extra machine no longer worth the money, and e good make you know this before you read wetin VPS dey cost per month or move the workload go dedicated server.

FAQ

Why I dey get different benchmark result every time I run am?

VPS dey share physical CPU, storage and network with other tenants, so wetin dem dey do for that moment go affect your result. Run vmstat 1 during the test and read the st column: sustained steal time records CPU time wey no dey available, but e fit show provider dey enforce the CPU allocation wey you buy, and e fit also come from host contention. Value wey pass 5% no prove the cause. The correct answer na better method, no be tuning. Run each test five or more times across different hours, then report the median together with the spread.

Why fio dey report millions of IOPS?

Almost always because --direct=1 dey missing. Without am, fio dey read through the kernel page cache. After the first pass, RAM dey serve the 2G test file, so na memory bandwidth you don measure. Add --direct=1 and make the test file bigger than any cache for the path. If --direct=1 fail after that, run df -hT . and check wetin the test file dey sit on. overlay type no be the problem by itself, because overlayfs dey pass O_DIRECT through to the storage underneath. Move the test file to real local storage only if the filesystem underneath no allow direct IO.

yabs.sh dey enough by itself?

For first look, yes. E dey run fio with four block sizes, iperf3 for both directions, and Geekbench, then e dey print one summary wey other people fit read. E no longer dey enough when you want know why one number be wetin e be, because you no fit change the flags for each test. Once yabs result look wrong, reproduce am directly with fio or sysbench and change one flag at a time.

Which one number fit predict how my application go feel?

Single core CPU speed and 4k random read latency, in that order, for most web and database workloads. Throughput figures dey look impressive but dem rarely decide anything, because typical request dey small. Quote the 99th percentile from the fio clat percentiles block instead of the average, because na the one slow request out of 100 wey user go notice.

I need install anything before I benchmark?

fio, sysbench and iperf3 dey all inside Ubuntu and Debian archives: sudo apt install -y fio sysbench iperf3. yabs.sh dey run for Bash and e need curl or wget for downloads, plus tar to extract Geekbench; without tar, e go skip Geekbench. E dey use fio and iperf3 when dem dey installed, and e dey download static binaries when dem no dey available. Delete every test file when you finish, because 2G fio file wey remain for 20G disk fit become somebody disk full alert weeks later.