How Often You Suppose Scrub Small ZFS VPS Pool?

Wetin ZFS scrub actually dey do

ZFS scrub dey read every allocated block for the pool, recompute the checksum, then compare the result with the checksum wey dey inside the parent block pointer. If the two no match, ZFS repair the block from any redundancy wey the pool get. Nothing else for ZFS fit do this work. Normal reads dey verify only the blocks wey you happen touch, so file wey you never open for two years go remain unverified until scrub read am.

Scrub no be fsck, the offline repair pass wey other filesystems need. No structural repair phase dey, because ZFS never leave the on-disk format for broken state: every write dey go new location, and uberblock, wey be the pool root pointer, dey update last. Scrub too no dey read the whole device. E dey read allocated blocks only. Na why almost empty pool fit scrub within minutes, while the same pool when e reach 80% full go take much longer.

Scrub dey run with the lowest I/O priority wey ZFS get. For Linux, zfs_vdev_scrub_max_active default na 2, so no pass two scrub reads dey in flight per vdev (virtual device, the group of disks wey ZFS dey treat as one unit). zfs_scrub_min_time_ms default na 750, the minimum time wey the sync thread dey spend on scrub work between transaction group flushes, the periodic commits wey ZFS dey batch writes into. For idle machine, scrub dey use the whole disk. Under load, e dey give way. For pool wey get one or two devices, nowhere dey for am to give way to, na why scheduling matter more here than for large chassis wey get sixty drives.

Why scrub pool wey no fit repair itself?

Na this sentence dey decide everything else for small pool. If redundancy no dey, scrub go detect corruption but e no fit fix am. One virtual disk for VPS na pool wey no get mirror and no get parity. ZFS go read the bad block, checksum go fail, e go count am for CKSUM column, name the file, then stop there, because second copy no dey to rebuild from.

Two partial exceptions dey wey you suppose know. By default, ZFS dey store extra copy of metadata (redundant_metadata=all) for different region of the device, so scrub fit repair damaged directory entry or block pointer even for one-device pool. Dataset wey get copies=2 dey keep two copies of its data blocks, but e go use double the space. None of dem fit survive if device go offline. The copies property documentation warn about exactly this: no build striped pool, set copies=2, then believe say redundancy dey.

So for single-device pool, scrub dey give you one thing: early and accurate notification. E turn silent corruption into filename for zpool status -v while your backup still get good version of that file. This one support backups; e no mean say you should avoid scrubbing. If you never settle the difference between point-in-time image and real off-box copy, start with why VPS snapshot no be backup, because scrub result only help if another place get intact copy.

Scrub wey find nothing na result too. E tell you say the data wey you wan trust dey intact, and na this you need know before restore or migration.

How often you suppose scrub small VPS pool?

Every month na the correct default, and na wetin the packages already expect. Debian and Ubuntu ship cron job wey scrub healthy pools on the second Sunday of every month. FreeBSD periodic system dey use threshold for days, and daily_scrub_zfs_default_threshold default na 35, wey the manual describe as five weeks.

Weekly scrubbing for busy small pool usually cost pass wetin e return. With one or two devices, the scrub dey compete for the same queue with your application, and no spare device dey to absorb am. For VPS, I/O allowance limited, so the reads wey scrub use na reads wey your database no fit use. Against this cost, weekly scrubbing fit give you at most three weeks earlier warning about fault wey you no fit repair anyway. This trade only make sense when scrub cheap.

Measure the time, then decide. Run one scrub by hand and monitor how long e take.

1. Run sudo zpool scrub tank for quiet evening and record the total time from zpool status.
2. If e finish well under one hour and the box dey idle overnight, weekly dey affordable.
3. If e run for many hours while the pool dey serve traffic, stay with monthly and allow the packaged job to manage am.
4. Measure the time again whenever the pool grow noticeably, because scrub duration dey follow allocated data, not disk capacity.

Any option wey you choose, write am down beside your other recurring server work. Scrub belong for the same list with package upgrades and log rotation: see monthly Linux server maintenance checklist.

Start, pause and stop a scrub

sudo zpool scrub tank
sudo zpool status tank

Pause and stop no be the same operation, and if you choose the wrong one, you fit lose hours of repeated work.

sudo zpool scrub -p tank
sudo zpool scrub tank
sudo zpool scrub -s tank

-p dey pause am. The pause state and progress dey sync to disk from time to time, so paused scrub go survive export or reboot: when pool come back, scrub still dey paused and dey wait for you. If you run zpool scrub again, e go continue from the last checkpoint wey dem write to disk. -s go stop the scrub instead, and the next scrub wey you start go begin from the start. Use -p when you need the disk back for one hour. Use -s when you want the scrub to end completely.

Two other flags dey worth knowing. -w go wait until scrub finish before e return, and na wetin you want inside script so the next step no start early. -e go scrub only the files wey get known data errors according to zpool status -v. This na the quick way to confirm say file wey you restore from backup don clean now.

ZFS dey run only one scrub or resilver at a time for each pool. Resilver na the rebuild wey follow after you replace device. Both operations dey use plenty I/O. If device dey resilver, your scrub go wait for im turn.

How to read zpool status while a scrub runs

Run sudo zpool status tank and read your own numbers instead of comparing dem with another person own. While scrub dey run, the scan: line get scanned figure, issued figure, total, repaired figure, percentage wey don complete, and estimated time remaining.

Scanned na the metadata phase: ZFS dey walk the block tree and collect the addresses wey e need read. Issued na the data phase: na the reads wey don actually go the device, arranged according to disk order. Sorted scrub na why these two counters dey, and issued na the one wey dey track real progress. For the beginning, scanned dey far ahead of issued, so the time estimate no mean much. Judge am after the first ten percent.

Repaired dey count bytes wey dem rewrite from a good copy. For pool wey no get redundancy, this one go remain zero no matter wetin scrub find. This na the earlier point wey we don restate as number wey you fit monitor.

Then read the columns for each device. READ and WRITE dey count I/O errors wey the device itself report. CKSUM dey count blocks wey fail checksum verification, and CKSUM na the column wey scrub dey meant to fill. Non-zero CKSUM for device wey look healthy na real problem: the data return, but e return wrong.

The last line na the verdict. errors: No known data errors mean say the pass succeed. Anything else mean say you need run sudo zpool status -v tank, wey go print the complete list of data errors since the last complete scrub, including the affected filenames. Restore those files from backup, run sudo zpool clear tank to reset the counters, then run scrub again. Every complete scrub rebuild this list, so if filename no dey there after full clean scrub, e truly don clear.

Which periodic scrub job dey run for your box?

No assume say one dey, and no assume say na only one dey. The mechanism dey differ by platform and package. The pool format dey identical everywhere, so e easy forget say the tools around am no be the same. the way ZFS dey ship for FreeBSD compared with Linux na the difference wey matter here.

For FreeBSD, the job dey inside the periodic system. Set these for /etc/periodic.conf:

daily_scrub_zfs_enable="YES"
daily_scrub_zfs_pools="tank"
daily_scrub_zfs_default_threshold="35"

daily_scrub_zfs_pools na space-separated list of pool names. If you leave am empty, e go scrub every pool. daily_scrub_zfs_default_threshold na the number of days between scrubs when no pool-specific threshold dey set. The manual give 35 as the default. The daily job dey run every day, but e go start scrub only after the threshold don pass.

For Linux, e depend on your distribution ZFS package, and some systems get both mechanisms at once. There are per-pool systemd timers, zfs-scrub-monthly@tank.timer and zfs-scrub-weekly@tank.timer, wey you enable one pool at a time. Debian and Ubuntu also ship /etc/cron.d/zfsutils-linux. E dey run a script wey scrubs every ONLINE pool on the second Sunday of each month. Check wetin you get before you add anything:

systemctl list-timers 'zfs-*'
ls /etc/cron.d/ | grep -i zfs
sudo zpool history tank | grep scrub

zpool history na the correct answer, because e records the scrubs wey the pool actually start, together with dates. If two scrubs dey happen every month, both mechanisms dey active and one of dem suppose go. To enable a timer:

sudo systemctl enable --now zfs-scrub-monthly@tank.timer

Free space matter pass any tunable

How long scrub go take for small pool depend on how much data dey allocated and how scattered the data be. If pool full too much, both things go worse.

OpenZFS guidance be say make pool free space stay above 10%. Below that point, metaslabs, wey be the chunks wey allocator dey work with, start to cross 4% free threshold. Then allocator switch from first-fit to best-fit. Best-fit dey use much more CPU. Write latency go rise, fragmentation go follow, and the next scrub go slow more, because the same amount of data now dey come as more and smaller reads.

So, the first thing to do no be tunable. Na to delete things. Old snapshots na the usual cause for ZFS box. After that, na Docker images and layers wey nobody don prune and kernel packages wey upgrades leave behind. Run zfs list -o space before you touch anything else, because e go separate space wey snapshots hold from space wey live data dey use.

Now, make we look the knobs briefly. For Linux, you fit read the current values:

cat /sys/module/zfs/parameters/zfs_scrub_min_time_ms
cat /sys/module/zfs/parameters/zfs_vdev_scrub_max_active

FreeBSD dey expose the same parameters through sysctl, so find your own with sysctl -a | grep scrub. If you raise them, scrub go finish sooner, but your application go slow down. If you lower them, the opposite go happen. For pool wey get one or two devices, no setting fit give you both, because na only one queue dey available to share. Knob rarely dey fix design problem. If monthly scrub dey cause problem, the honest meaning be say pool full too much or device slow too much. Tunable go only shift the problem.

Scrub time na preview for your resilver

Resilver dey do the same scan like scrub: e reads the allocated blocks, verify dem, then write the missing ones to the replacement device. So, the time your scrub takes na the closest honest preview of how long rebuild go take, and how long the pool go run with reduced redundancy during that period.

ZFS dey schedule resilver work more aggressively than scrub work, so rebuild usually finish faster than scrub for the same pool. Treat your scrub time as conservative upper limit. If scrub dey take nine hours, plan for rebuild window wey dey around that length, and understand say if another device fail inside that window, you lose the pool. Na this be the practical reason to use mirrored pairs instead of one wide raidz group, because raidz, wey be the parity layout ZFS dey use instead of RAID 5, rebuilds by reading every surviving device.

For pool wey get only one device, resilver no dey happen at all. If the device fail, the pool fail with am. Your recovery time na your restore time, so measure the restore instead. Restore wey you never run no be recovery plan.

Wetin change when you rent the disk

For VPS, the block device na virtual one. The hypervisor go present one volume, and underneath e fit get local NVMe, or replicated network volume wey get im own parity. Two things follow from this for scrubs.

First, the platform redundancy dey hidden from ZFS, and ZFS no fit use am. If the platform repair media error below your instance, ZFS no go ever see the problem. If the platform pass wrong block up to you, ZFS go catch am but e no fit fix am, because the correct copy dey for the other side of that boundary.

Second, normally you no fit read SMART (self-monitoring, analysis and reporting technology) data for the device underneath virtual disk. So the early warnings wey disk health monitoring for VPS depend on fit no dey available at all. The CKSUM counter from your scrubs become the main signal wey you control.

If you want ZFS to repair instead of only report, the pool need more than one device inside the same instance. This na plan decision, no be tuning decision. Choosing storage VPS instead of regular VPS go give you the capacity, but whether e go give you two independent devices depend on the plan. Run lsblk and confirm before you build mirror on wetin fit turn out to be two slices of one volume. We dey rent Linux and FreeBSD servers, no be managed ZFS appliance. So na you go run the scrub schedule and backups. Na this be the trade-off: full control of the pool, and full responsibility for maintaining am.

FAQ

How often should I scrub a ZFS pool on a VPS?

Monthly dey okay for most small pools, and e match wetin the packages already dey do: second-Sunday cron job for Debian and Ubuntu, and 35 day default threshold for FreeBSD periodic system. Weekly make sense only after you don time one scrub and confirm say e dey finish quickly for box wey no dey busy. For pool wey busy and get one or two devices, weekly scrubbing dey use real application I/O every week, but e only give you few weeks of earlier warning.

Is scrubbing a single-disk ZFS pool pointless?

No, as long as you understand wetin e fit do. Without redundancy, scrub fit detect corruption but e no fit repair am, except metadata, because ZFS dey keep one extra copy by default. Wetin you get na named list of damaged files for zpool status -v, early enough to restore dem while good copy still dey somewhere else. The correct response na better backups, because scrub go tell you exactly which file you need restore.

Can I pause a ZFS scrub and finish it later?

Yes. zpool scrub -p tank dey pause am, and e dey write the pause state and progress to disk periodically, so scrub remain paused across export or reboot. Run zpool scrub tank again to resume from the last checkpoint. No use zpool scrub -s tank for this: -s dey stop the scrub, and the next one go start again from the beginning.

Why is my ZFS scrub so slow, and can I speed it up?

Scrub time dey follow allocated data and fragmentation, not disk capacity. Pool wey pass 90% full dey slow because metaslabs wey get less than 4% free dey push allocator from first-fit to best-fit, and the fragmentation wey follow dey turn scrub into plenty small reads. Freeing space normally dey help pass any tunable. You fit increase zfs_scrub_min_time_ms or zfs_vdev_scrub_max_active to give scrub bigger share of the queue, but for pool wey get one or two devices, that share go come directly from your application.