Disk health monitoring on a VPS

What disk health monitoring on a VPS can actually see

Disk health monitoring on a VPS starts with a fact most guides avoid: the disk is not yours. Your guest sees a virtual block device. The physical drive, and every counter stored on it, belongs to the host. smartctl /dev/vda does not fail because you typed the command wrong. It fails because nothing behind that device can answer the question.

SMART (self-monitoring, analysis and reporting technology) is a table of counters kept on the drive itself: reallocated sectors, pending sectors, power-on hours, media errors. Reading that table needs a path for ATA or NVMe (non-volatile memory express) commands to reach real hardware. A paravirtual disk does not provide one, so the guest gets storage with the telemetry stripped off.

A tenant monitors effects, not hardware. Four signals are visible from inside the guest: I/O (input/output) errors in the kernel log, a filesystem that remounts read-only, latency that drifts upward, and space that runs out. All four can be alerted on today, and all four show up before a user complains. Set those up first. The division of responsibility comes at the end, because it changes where you should spend effort.

Prove what your own server exposes

Do not assume which case you are in. Look, then read the section that matches.

sudo apt update && sudo apt install -y smartmontools nvme-cli
lsblk -o NAME,TYPE,SIZE,MODEL,TRAN
sudo smartctl -a /dev/vda

virtio-blk, the usual KVM (kernel-based virtual machine) disk. The device is /dev/vda and smartctl stops before it sends anything:

/dev/vda: Unable to detect device type
Please specify device type with the -d option.

virtio-blk is a paravirtual transport with no ATA or SCSI command set behind it, so there is no channel to carry a SMART request. -d sat and -d scsi fail the same way, because the transport is the problem and not the flag.

An emulated SATA or SCSI disk. The device is /dev/sda and smartctl gets far enough to identify it. The model line reads QEMU HARDDISK. That string answers the question by itself: you are reading a device the emulator invented, and it reports no usable SMART capability.

An NVMe namespace. sudo nvme smart-log /dev/nvme0n1 returns a full log, which is where people get fooled. Check the controller identity first with sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|sn)'. A model number that names a network storage product means the controller is software, so percentage_used and media_errors describe that emulation rather than the flash under your data. If you want to know what your storage really is, verify the NVMe disk on Linux instead of trusting the plan description.

A container, such as LXC (Linux containers) or OpenVZ. You have no block device of your own. lsblk shows the host's devices or nothing at all, and smartctl is refused because the container does not hold CAP_SYS_RAWIO:

Smartctl open device: /dev/sda failed: Permission denied

One warning about the case where it does work. If smartctl on a VPS returns a full attribute table, read the serial number before you act on it. Some hosts expose a passthrough device node, and those counters belong to hardware shared by every tenant on that machine. A rising Reallocated_Sector_Ct there is a support ticket. It is not a statement about your data.

Signal 1: I/O errors in the kernel log

This is the highest value signal a tenant has, and it needs no agent.

sudo journalctl -k -p err -b
sudo journalctl -k --since "7 days ago" | grep -iE 'i/o error|remount|ext4-fs error|buffer i/o'

A failed request from the virtual disk looks like this:

blk_update_request: I/O error, dev vda, sector 2101248 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0

The block layer asked the host for a write and the host returned a failure. On a VPS that is rarely a dying flash cell. It is usually the host storage layer or the network path to network attached storage, so it is a provider side event. Copy the timestamp, the device name and the sector into your ticket, because those are what a storage team can match against their own logs.

The ext4 sequence that matters most is this pair:

EXT4-fs error (device vda1): ext4_journal_check_start:83: comm cron: Detected aborted journal
EXT4-fs (vda1): Remounting filesystem read-only

The second line is the one that hurts, because the machine stays up. It answers ping, it answers SSH, and every write fails. A plain HTTP check keeps passing while your application throws an error on every request.

XFS shuts the filesystem down instead:

XFS (vda1): metadata I/O error in "xfs_trans_read_buf_map+0x1c0/0x2e0" at daddr 0x2 len 1 error 5
XFS (vda1): I/O Error Detected. Shutting down filesystem

journalctl -k reads only the current boot unless the journal is stored on disk, and many images ship a volatile journal that lives in RAM. Turn on persistence, or the evidence vanishes at exactly the reboot you will perform while troubleshooting.

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --list-boots

After your next reboot, journalctl --list-boots should list more than one boot. Even with persistence on, a filesystem that has gone read-only cannot record what happened next, which is the honest argument for shipping logs off the box.

Signal 2: catching a read-only remount

Make the failure loud before you try to detect it.

findmnt -no SOURCE,FSTYPE,OPTIONS /

Look for errors=remount-ro in the options. Ubuntu and Debian cloud images set it in /etc/fstab, so a metadata error takes the filesystem read-only instead of continuing over damage. If it is missing, add it to the root entry in /etc/fstab, or set it in the superblock with sudo tune2fs -e remount-ro /dev/vda1. A loud stop beats quiet corruption.

A mount flag is not proof. Probe by writing:

touch /var/tmp/.disk-probe

On a read-only root that prints exactly:

touch: cannot touch '/var/tmp/.disk-probe': Read-only file system

Use /var/tmp, not /tmp. On most images /tmp is a tmpfs held in memory, so a successful write there proves nothing about your disk.

Wrap the write test together with a space check, and send a heartbeat only when every check passes:

sudo tee /usr/local/sbin/disk-probe >/dev/null <<'EOF'
#!/bin/sh
set -eu
probe=/var/tmp/.disk-probe
echo ok > "$probe"
test "$(cat "$probe")" = ok
rm -f "$probe"
used=$(df --output=pcent / | tail -n1 | tr -dc '0-9')
test "$used" -lt 90
inodes=$(df --output=ipcent / | tail -n1 | tr -dc '0-9')
test "$inodes" -lt 90
curl -fsS --max-time 10 "https://status.example.com/api/push/REPLACE_TOKEN?status=up&msg=OK" >/dev/null
EOF
sudo chmod 755 /usr/local/sbin/disk-probe
sudo /usr/local/sbin/disk-probe && echo probe-ok

probe-ok on that last line means the whole chain works. set -eu makes any failed check exit non-zero before the curl line runs, so no heartbeat goes out. That inversion is the point: the monitor turns red because nothing arrived, and a server that cannot write cannot be trusted to describe its own problem. Reads still work on a read-only filesystem, so the script itself still starts.

Run it from a systemd timer.

# /etc/systemd/system/disk-probe.service
[Unit]
Description=Disk writability and space probe

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/disk-probe

# /etc/systemd/system/disk-probe.timer
[Unit]
Description=Run the disk probe every five minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now disk-probe.timer
systemctl list-timers disk-probe.timer
journalctl -u disk-probe.service -n 20 --no-pager

systemctl list-timers should show the unit with a NEXT time under five minutes away. A failed run appears in journalctl -u disk-probe.service with the shell's own error text, so you can tell a read-only filesystem from a full one without logging in.

That push URL is an Uptime Kuma push monitor. Create a monitor of type Push, copy its token into the script, and set the monitor's heartbeat interval a little longer than the timer interval so one slow run does not page you at 03:00. If you have no status page yet, a self-hosted Uptime Kuma instance is the cheapest place to put this check.

Two honest limits. The probe confirms that a write was accepted, not that the bytes reached durable storage, because the read back can be served from the page cache. And it runs on the machine it watches, so a fully wedged server goes silent rather than reporting a diagnosis.

Signal 3: latency and throughput trends

sudo apt install -y sysstat
iostat -xdz 5 3

Read r_await and w_await first. They are the average milliseconds a read or a write took, including time spent waiting in the queue. Read aqu-sz next, the average number of requests in flight. Ignore %util on a virtual disk: it only means the queue was not empty, and a device that serves many requests in parallel sits near 100 percent while it is nowhere near its limit. await is the number that tracks what users feel.

Absolute values matter less than your own baseline, so record a quiet hour and keep it. /proc/diskstats is the raw source if you would rather collect the counters yourself.

For a deliberate measurement:

sudo apt install -y fio
fio --name=readlat --filename=/var/tmp/fio.probe --size=512M --rw=randread --bs=4k --iodepth=1 --direct=1 --runtime=30 --time_based --group_reporting
rm -f /var/tmp/fio.probe

Read the clat percentiles block, in particular the 99th. --direct=1 skips your page cache. It does not skip the host's cache, so the result describes the whole path from your process down to the platform's storage. Run it while the server is idle, since it competes with your own workload.

Rising await with no errors in the kernel log is usually not a failing drive. It is contention on the host, the storage version of CPU steal time from a noisy neighbour. If it returns at the same hour every day and your ticket comes back clean, the answer is a plan whose I/O is not shared the same way, which is the case for a storage VPS over a regular VPS when the workload is disk bound.

Signal 4: filesystem checks you can run while mounted

ext4 keeps an error counter in the superblock, and it survives reboots even when your logs do not.

sudo dumpe2fs -h /dev/vda1 2>/dev/null | grep -iE 'filesystem state|error count|first error|last error'

A healthy filesystem prints Filesystem state: clean and FS Error count: 0. clean with errors and a non-zero count mean the kernel hit a metadata error at some point, even if nobody noticed and the log has rolled away. That one command belongs in a weekly check.

You cannot fsck a mounted root filesystem, and e2fsck -n on a live filesystem reports problems that are only the data changing underneath it. To force a real check, add fsck.mode=force fsck.repair=yes to the kernel command line for one boot from your provider's console. systemd-fsck then runs the check before the root is mounted read-write.

XFS has no online check. xfs_repair -n /dev/vda1 refuses to run against a mounted filesystem, so it belongs in rescue mode. XFS makes up for that by being loud: it shuts the filesystem down on a metadata error instead of continuing.

On Btrfs the counters are built in and persistent.

sudo btrfs device stats /
sudo btrfs scrub start -B /

write_io_errs or corruption_errs above zero is a real event, and the counters keep their values across reboots until you reset them. scrub re-reads every block and verifies its checksum, which is the closest thing to a media test available on a virtual disk. It is heavy on I/O, so schedule it for a quiet hour.

Signal 5: free space, including the parts df hides

Running out of space breaks a server the same way a bad disk does, and it happens far more often.

df -h /
df -i /
sudo du -xh --max-depth=1 / | sort -h | tail -n 20

No space left on device while df -h shows free space means you ran out of inodes rather than bytes, and df -i shows IUse% at 100 percent. Millions of small files in a cache directory or a mail spool cause this, and deleting large files does not help.

Space that does not come back after a delete is usually a deleted file still held open by a running process. sudo lsof +L1 lists files whose link count has reached zero. Restarting the process that holds one releases the space.

The journal is a common quiet consumer. journalctl --disk-usage reports what it holds. Cap it with SystemMaxUse=200M in /etc/systemd/journald.conf followed by sudo systemctl restart systemd-journald, and reclaim the space now with sudo journalctl --vacuum-size=200M.

One case looks like a bug and is not. On thin provisioned host storage the host's pool can fill while your df still shows free gigabytes. Your writes then fail with I/O errors in the kernel log and no space warning anywhere inside the guest. Errors without a full filesystem is a combination worth a ticket the same hour.

Wiring the signals into a metrics agent

A push probe answers yes or no. Trends need a metrics agent, and Prometheus node_exporter already exports everything above with no extra configuration. The metric names to build on:

• node_filesystem_readonly goes to 1 when a mount is read-only, which is your remount alarm.
• node_filesystem_avail_bytes and node_filesystem_files_free cover bytes and inodes separately.
• node_disk_io_time_seconds_total and node_disk_read_time_seconds_total give you busy time and latency as counters you can graph.

Two rules catch the cases that actually page:

- alert: FilesystemReadOnly
  expr: node_filesystem_readonly{fstype!~"tmpfs|overlay"} == 1
  for: 2m
- alert: FilesystemFillingUp
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*24*3600) < 0
  for: 30m

The second rule fires when the current trend reaches zero within four days, so you get warned days ahead instead of at 95 percent full, when you have minutes.

Who is responsible for what

Your provider owns the physical drives. They read SMART, they run the array, and they replace a drive with rising reallocated sectors, usually without telling you, because the array absorbs the failure. That is what RAID 10 under your VPS is for: a dead drive becomes a rebuild instead of an outage. You cannot see any of it, and paying for that abstraction is most of the point of renting a virtual server.

You own your data, and drive telemetry would not protect it anyway. The events that actually destroy tenant data are a mistaken rm, a bad deploy, an intruder with your SSH key, and a platform incident that takes the array with it. SMART attributes predict none of those.

So a tenant's real protection is a backup that lives off the server and a restore you have performed yourself. Provider snapshots are convenient and they sit on the same platform as the thing they protect, which is why snapshots and backups are different protections. Put a drill in the calendar: once a quarter, restore the newest backup into a fresh VPS, start the application, and write down how long it took. That number is your real recovery time. The first drill is always slower than anyone guessed.

When SMART does apply to you

Guides that teach smartctl are correct, and they apply the moment the hardware is really yours:

• A dedicated or bare metal server, where sudo smartctl -a /dev/sda returns the full attribute table and smartd can mail you when an attribute changes.
• Storage plans that pass a physical disk through to the guest. Providers document this explicitly, because it is a selling point.
• Hardware you own, at home or in rack space you rent.
• A disk behind a RAID controller, reachable with sudo smartctl -a -d megaraid,0 /dev/sda, or a USB enclosure with -d sat.

On real NVMe, sudo smartctl -a -d nvme /dev/nvme0 and sudo nvme smart-log /dev/nvme0n1 report critical_warning and percentage_used from the drive itself. On real SATA, the attributes that predict failure are Reallocated_Sector_Ct (5), Current_Pending_Sector (197), Offline_Uncorrectable (198) and Reported_Uncorrect (187). Any of them moving off zero means plan a replacement. Large scale drive studies keep landing on that same short list, and most other attributes are noise.

Run the daemon rather than checking by hand.

sudo systemctl enable --now smartd
sudo smartctl -t short /dev/sda
sudo smartctl -l selftest /dev/sda

The self-test log should show Completed without error for the run you just started. Ubuntu and Debian ship /etc/smartd.conf with a DEVICESCAN line, current as of August 2026, so the daemon picks up every disk it can see and mails root on a change. None of that works on a virtual disk, which is why the rest of this guide exists.

FAQ

Why does smartctl not work on my VPS?

Because the disk is virtual. On a KVM guest using virtio-blk, smartctl -a /dev/vda prints /dev/vda: Unable to detect device type, since a paravirtual disk carries no ATA or SCSI command channel for a SMART request to travel down. On an emulated disk you reach a device whose model reads QEMU HARDDISK, with no usable SMART data behind it. Inside a container, smartctl is refused outright for lack of CAP_SYS_RAWIO. None of these is a misconfiguration, and no -d flag fixes them.

How do I know if my VPS disk is failing?

Watch effects rather than hardware. Check sudo journalctl -k -p err -b for blk_update_request: I/O error lines and for Remounting filesystem read-only. Run sudo dumpe2fs -h /dev/vda1 | grep -i 'error count' to find errors the logs have already lost. Track r_await from iostat -xdz 5 against a baseline you recorded when things were healthy. On a VPS an I/O error usually means a host storage problem rather than a dying drive, so it belongs in a support ticket with the timestamp and sector attached.

What should I alert on for VPS disk health?

Four alerts cover it. A read-only mount, from node_filesystem_readonly == 1 or a write probe that fails. Free space and free inodes trending toward zero. Any kernel I/O error in the last interval. A heartbeat from the server, so that silence pages you when the box stops answering. Skip anything derived from SMART, because on a virtual disk those values are either missing or describe the hypervisor's emulation.

Why did my filesystem remount read-only?

ext4 mounted with errors=remount-ro does this deliberately when it hits a metadata error: it stops writing rather than continuing over damage. The trigger sits in the kernel log just above the remount line, usually an EXT4-fs error about an aborted journal after the underlying device returned an I/O error. Remounting read-write without checking the filesystem hides the symptom and keeps the cause. Capture the log, then check the filesystem unmounted from rescue mode with e2fsck -fy /dev/vda1.

Can I ever read SMART data on a virtual server?

In specific cases, yes. Dedicated and bare metal servers give you real attributes. So do storage plans that pass a physical disk through to the guest, and any host you own yourself. Some platforms present an NVMe controller to the guest and nvme smart-log returns a log, so run sudo nvme id-ctrl /dev/nvme0 first: a model number naming a network storage service means those counters come from a software controller. And where a passthrough node does expose real counters on a shared machine, they describe hardware shared with other tenants, so the only useful action is a support ticket.