Fix a degraded ZFS pool and replace the disk

How to fix a degraded ZFS pool, in short

A degraded ZFS pool still serves your data, but one of its vdevs has a member that is missing, offline or failing. If a member is missing or offline, that vdev now has less redundancy than you built it with. A vdev (virtual device) is one of the mirror or raidz groups that make up the pool. To fix it, first confirm you have a current copy of the data on another machine. Then find out what is wrong with the device. If the problem was the connection to the disk, bring the device back with zpool online. If the disk is dead, swap it out with zpool replace. Finish with a scrub, so ZFS checks every block before you call the pool healthy again.

Every command here talks to the ZFS kernel module. Run them on your own server as root or with sudo. The pool in the examples is called tank. Replace that with your pool's name.

Which OpenZFS version are you running?

The commands below are the same on Ubuntu 24.04 and 26.04, but the OpenZFS release is different. As of 17 October 2026, packages.ubuntu.com lists zfsutils-linux 2.2.2 for Ubuntu 24.04 (build 2.2.2-0ubuntu9.5 on amd64). For Ubuntu 26.04 it lists 2.4.1 (build 2.4.1-1ubuntu5.1 on amd64 and arm64). Security updates change the build number, so check your own server:

apt policy zfsutils-linux
zfs version

This guide uses three options that some readers may not know: zpool status -e, zpool replace -s and zpool online -e. The last two appear in the OpenZFS 2.2.2 man pages that Ubuntu 24.04 ships, and in later ones. zpool status -e is not in that 2.2.2 build; it first appears in OpenZFS 2.2.3. If you are unsure, run man zpool-replace on your server. It describes the exact version you have.

What DEGRADED means next to FAULTED, UNAVAIL, REMOVED and OFFLINE

The OpenZFS zpoolconcepts(7) man page says a pool's health is one of three states: online, degraded or faulted. Single devices can show more states than that. Here is what each word in the STATE column of zpool status means:

• ONLINE: the device or vdev is working normally.
• DEGRADED: on a vdev, one or more of its devices is offline, faulted or degraded, but in the man page's words, "Sufficient replicas exist to continue functioning." On a single device, DEGRADED usually means its checksum errors or slow I/Os have passed acceptable levels, as "an indication that something may be wrong." The man page adds: "ZFS continues to use the device as necessary." A device with too many I/O errors is also marked DEGRADED instead of FAULTED when too few replicas remain to fault it. Your data is still readable.
• FAULTED: "Insufficient replicas exist to continue functioning." On a top-level vdev, this means ZFS can no longer serve the data stored there.
• UNAVAIL: "The device could not be opened." If the pool was imported while the device was missing, zpool status shows the device as a long numeric identifier instead of a path. The man page explains this: the path was never right in the first place.
• REMOVED: the device was physically removed while the system was running. Whether ZFS can detect this depends on the hardware, so not every platform reports it.
• OFFLINE: an administrator took the device offline with zpool offline. ZFS does not read from or write to an offline device.

A DEGRADED pool is a pool with at least one DEGRADED vdev. The pool keeps working, and that is the reason people leave it alone for too long.

Why one more failure can lose the data

The man page states the rule directly. A mirror with N disks "can withstand N-1 devices failing, without losing data." A raidz group with P parity disks can survive P failures, and raidz1 has one parity disk.

So count how many failures the vdev can still survive. A two-way mirror has N = 2, so it survives one failure. If one member is already unavailable, offline or faulted, the next failure in that mirror leaves no copy to read from. A raidz1 vdev has P = 1, so a raidz1 vdev with one member unavailable has nothing left either. One more failure turns the vdev FAULTED, and the data on it is lost. A raidz2 vdev or a three-way mirror with one member gone can still survive one more failure. If you are still choosing a layout, RAID 10 and striped mirrors for VPS storage compares the options.

The word DEGRADED alone does not tell you which case you are in. A vdev is also DEGRADED when every member is present but one has passed the checksum error or slow I/O threshold, and ZFS still reads from it. That vdev has all its members, but one of them is not reliable. Look at the state of each device under the vdev in zpool status. A member shown as UNAVAIL, REMOVED, OFFLINE or FAULTED has stopped taking part. A member shown as DEGRADED is still in use, either because its errors are a warning sign or because ZFS has too few replicas to fault it. Either way, investigate it before anything else fails.

The repair itself puts extra load on the disks. A resilver is ZFS's word for rebuilding a device from the surviving copies, and it has to read the remaining members. If one of those disks is also weak, the resilver is when it is most likely to fail.

Step 1: confirm a current copy exists off the box

Do this before you touch any disk. If you already replicate with zfs send, log in to the backup machine and look at the newest snapshot it holds:

zfs list -t snapshot -o name,creation -s creation backup/tank | tail -n 5

The dataset name backup/tank is only an example. Use the dataset your replication writes to. The newest creation time tells you how much data you would lose if the repair goes wrong.

If there is no recent copy, make one now, while the pool still has a working copy of every block. Take a recursive snapshot and send it to another machine. Sending ZFS snapshots to a storage VPS with zfs send and receive covers the full procedure, including incremental sends for later.

sudo zfs snapshot -r tank@before-repair

A local snapshot is not a backup on its own, because it lives on the same disks that are failing.

Step 2: read what zpool status is telling you

sudo zpool status -x
sudo zpool status -v tank
sudo zpool status -e tank
sudo zpool status -LP tank

Here is what each option does:

• -x shows only pools that have errors or are otherwise unavailable.
• -v adds a list of every data error since the last complete scrub. If the list names files, those files are damaged and you need to restore them from your backup.
• -e shows only unhealthy vdevs, which is easier to read on a large pool. It needs OpenZFS 2.2.3 or later, so skip it on Ubuntu 24.04's 2.2.2 build.
• -L resolves symbolic links, so you see the current kernel device name (sdb, vdc) behind each path. -P prints full paths instead of only the last part.

Write down the device name exactly as zpool status prints it. Every later command takes that name. If the device shows as UNAVAIL, use its numeric identifier instead.

The status output also names an OpenZFS message ID, with a link to a page that explains it. Two IDs you will often see are ZFS-8000-2Q and ZFS-8000-4J. The first page says "A device in a replicated configuration could not be opened." The second says "A device could not be opened due to a missing or invalid device label." Both pages list the recovery steps that this guide follows.

Next, find out which real disk each device name points to:

lsblk -o NAME,SIZE,SERIAL,MODEL
ls -l /dev/disk/by-id/

/dev/disk/by-id/ holds stable names, usually built from the disk's model and serial number. Each one is a link to the current sdX or vdX name. Use these links to work out which physical disk, or which provider volume, the failed device really is.

Is it a path problem or a dead disk?

The state word alone does not tell you this. A disk with a loose cable and a disk with dead electronics can both show UNAVAIL. Decide from what happened before the failure and from what the hardware shows now.

Treat it as a temporary connection problem when one of these is true:

• The device dropped out right after you moved a cable or reseated a controller.
• A VPS block volume was detached, in the provider panel or by an API call, and is now attached again.
• The server rebooted, and the pool was built on short names such as /dev/sdb. A short name can point at a different disk after a reboot.
• The disk shows up again in lsblk, and its health data looks clean.

Treat it as a dead disk when the device does not come back in lsblk. The same goes for SMART data that shows reallocated or pending sectors, and for error counters that start rising again soon after you clear them. If you do not watch disk health yet, monitoring disk health with SMART on a server shows what to read.

A member shown as DEGRADED is still in use. Check its cabling and SMART data. If both look clean, clear the counters and scrub as shown below, and replace the disk if the errors come back.

Fix a temporary problem: zpool online, zpool clear, then scrub

Once the device is reachable again, tell ZFS to use it:

sudo zpool online tank DEVICE
sudo zpool status tank

DEVICE is the name or identifier from zpool status. ZFS then resilvers the device. A resilver only examines the data that ZFS knows is out of date. After a short outage, that means it has less work to do than a full rebuild. Wait until zpool status no longer shows a resilver in progress.

If the device is ONLINE but still shows read, write or checksum errors from the outage, clear the counters:

sudo zpool clear tank

With no device name, zpool clear clears all device errors in the pool. Add a device name to clear only that device. If the pool was suspended, zpool clear also brings it back online, as long as ZFS can reach the devices. It does not repair anything. Always follow it with a scrub, which checks the checksum of every block:

sudo zpool scrub -w tank
sudo zpool status -v tank

The -w option makes the command wait until the scrub completes. ZFS will not start a scrub while a resilver is running. If the command refuses to start, wait for the resilver to finish first. A clean result shows every device ONLINE with zero errors, and no data errors listed.

Then watch whether the problem comes back. Run sudo zpool status -x again over the next few days. If a device drops out a second time, treat it as a dead disk. Regular scrubs find silent damage before a disk failure exposes it, and a scrub schedule for small ZFS pools covers how often to run them.

The device came back under a different name after a reboot

If the pool was created with /dev/sdX names, a reboot can give those names to different disks. ZFS then cannot find a member where it expects it. This fix works for a data pool, not the pool your system boots from. Export the pool, then import it again using the stable names:

sudo zpool export tank
sudo zpool import -d /dev/disk/by-id tank
sudo zpool status -P tank

Exporting unmounts every dataset in the pool, so stop the services that use the pool first. After the import, zpool status -P should show /dev/disk/by-id/ paths. These names do not depend on the order in which the kernel finds the disks.

Replace a dead disk with zpool replace

Refer to the new disk by its /dev/disk/by-id/ name. In the examples, OLD_DISK_ID is the failed device exactly as zpool status shows it. /dev/disk/by-id/NEW_DISK_ID is the new disk.

1. Take the failed disk offline

sudo zpool offline tank OLD_DISK_ID

While a device is offline, ZFS does not try to read from it or write to it. The OpenZFS man page does not say whether a plain offline lasts through a reboot. If the server reboots during the repair, check zpool status again. If you want the offline to be temporary, use zpool offline -t: the device then returns to its previous state when the server reboots.

2. Swap the hardware

On a physical server, power down first if your chassis has no hot-swap bays, then fit the new disk. On a VPS, attach a new block volume in your provider's panel. Make it at least as large as the old one. Attaching a block storage volume to a VPS covers how to attach a volume and find the new device. Then confirm that the kernel sees the new disk:

lsblk -o NAME,SIZE,SERIAL,MODEL
ls -l /dev/disk/by-id/

The man page requires the new disk to be at least as large as the smallest device in the mirror or raidz vdev. Compare the SIZE column before you continue.

3. Run zpool replace

If the new disk has a new name of its own, use the two-argument form:

sudo zpool replace tank OLD_DISK_ID /dev/disk/by-id/NEW_DISK_ID

If you put the new disk into the same physical slot and it took over the old path, the one-argument form is enough. Without a second argument, the new device defaults to the old device name:

sudo zpool replace tank OLD_DISK_ID

The man page describes zpool replace as "equivalent to attaching new-device, waiting for it to resilver, and then detaching device." This means the old device stays in the pool until the resilver finishes. Starting a replace also cancels any scrub that is running.

Four options are worth knowing:

• -w makes the command wait until the replacement completes before it returns.
• -f forces ZFS to use a disk that appears to be in use. Use it only after lsblk shows you that the disk holds nothing you need.
• -s rebuilds a mirror sequentially, which restores redundancy as fast as possible. It does not verify checksums during the rebuild, so ZFS starts a scrub when it finishes. Sequential rebuild does not work on raidz.
• -o ashift=N sets the only property that replace accepts. The ZFS settings you cannot change later explains why ashift matters for a new disk.

4. Follow the resilver

watch -n 60 sudo zpool status tank

While a resilver runs, zpool status shows how far it has got as a percentage, plus an estimated time to completion. The man page says these figures are only approximate. The real time depends on how much data the pool holds and how fast the surviving disks are. Read the estimate on your own system instead of trusting a figure from somewhere else.

5. Finish with a scrub

sudo zpool scrub -w tank
sudo zpool status -v tank

The repair is complete when four things are true. Every device is ONLINE. The error counters are zero. The old device is gone from the list. No files show data errors.

The mirror alternative: zpool attach, then zpool detach

On a mirror, if the old disk still works at all, you can add the new disk first and remove the old one afterwards. Done this way, redundancy never drops below its current level.

sudo zpool attach tank EXISTING_DISK_ID /dev/disk/by-id/NEW_DISK_ID
sudo zpool status tank
sudo zpool detach tank OLD_DISK_ID

zpool attach adds the new device to the mirror that EXISTING_DISK_ID belongs to. While the new disk resilvers, a two-way mirror becomes a three-way mirror. Run zpool detach only after zpool status shows that the resilver is complete. ZFS refuses a detach if no other valid copy of the data would remain. If you plan to reuse the old disk in this pool later, the man page suggests zpool offline instead of detach.

Does a ZFS pool rebuild on its own?

Not by default. The OpenZFS 2.2 and 2.4 man pages both list the pool property autoreplace as off by default. When it is off, an administrator must start every replacement with zpool replace. When it is on, ZFS acts on a new device that appears in the same physical location as a former pool member. It formats that device and uses it as the replacement automatically.

A hot spare is a disk that belongs to the pool but sits unused until an active device fails. Then the spare takes the failed device's place. It keeps that role until you replace the original device, and then it becomes an available spare again. Spares cannot replace log devices.

sudo zpool get autoreplace tank
sudo zpool add tank spare /dev/disk/by-id/SPARE_DISK_ID
systemctl status zfs-zed

The man page states: "Autoreplace and autoonline require the ZFS Event Daemon be configured and running." On Ubuntu that daemon comes from the zfs-zed package and runs as zfs-zed.service. If that unit is not active, do not count on any automatic repair, including spares. Check zpool status yourself.

Can mirror members be different sizes?

Yes, but the extra space on the larger disk goes unused. The man page says a mirror of N disks of size X holds X bytes. It also says a replacement must be at least as large as the smallest member. Taken together, this means the smallest member sets the size of the mirror. Use disks of the same size when you can.

Grow the pool onto bigger disks: autoexpand and zpool online -e

You can grow a mirror or raidz vdev by replacing each member with a larger disk, one disk at a time. Wait for each resilver to finish before you replace the next disk. Each replacement leaves the vdev with less redundancy until its resilver completes.

The new space appears only after every device in the vdev is larger. To use it, either turn on autoexpand (off by default) before the last replace, or claim the space by hand:

sudo zpool set autoexpand=on tank
sudo zpool online -e tank /dev/disk/by-id/NEW_DISK_ID
sudo zpool list tank

zpool online -e expands a device to use all of its space. In a mirror or raidz, you must expand every member before the pool can use the new space. zpool list should then show the larger size.

A note on FreeBSD device names

The same zpool commands work on FreeBSD, but disks have different names there. The FreeBSD Handbook uses adaN names for SATA disks and daN names for SCSI and SAS disks. Its examples build pools on GPT partitions such as ada1p1. Its replacement example takes ada1p1 offline, powers down, swaps the disk, and then runs zpool replace storage ada1p1. No second device name is needed there, because the new disk has the same device name as the old one. These names can change when you change the controller or the cabling, so confirm which physical disk failed before you pull one out. The Handbook also says that autoreplace is off by default. The FreeBSD daemon that acts on autoreplace, zfsd, is not enabled by default either. Running ZFS on FreeBSD and Linux covers the other differences between the two systems.

FAQ

What does DEGRADED mean in zpool status?

It means at least one vdev in the pool has a member that is missing, offline or failing, but enough copies of the data remain for the pool to keep working. Your data is still readable. If one member of a two-way mirror or raidz1 vdev is unavailable, that vdev has no redundancy left, and the next failure in it loses data. A member can also show DEGRADED after too many checksum errors or slow I/Os while ZFS keeps using it. Check the state of each device, confirm you have a copy on another machine, then repair the pool now.

Should I use zpool clear or zpool replace?

Use zpool online and zpool clear when the disk itself is fine and the problem was the connection to it. A loose cable is one example. A block volume that was detached and then attached again is another. Use zpool replace when the disk is dead or keeps producing errors. zpool clear only resets the error counters and does not repair anything. Follow it with zpool scrub, and watch whether the errors come back.

Will ZFS replace a failed disk automatically?

Only if you set it up. The autoreplace pool property is off by default, so you normally have to run zpool replace yourself. With autoreplace=on, or with a hot spare added through zpool add tank spare, ZFS can act on its own. Autoreplace needs the ZFS Event Daemon to be running. On Ubuntu, check that zfs-zed.service is active.

How long does a ZFS resilver take?

There is no fixed answer. It depends on how much data the pool stores and how fast the remaining disks are. While the resilver runs, zpool status prints the percentage done and an estimated time to completion. The OpenZFS man page says these figures are only approximate, so check the estimate again as the resilver goes on.

Can I replace a disk with a bigger one?

Yes. The new disk must be at least as large as the smallest member of the vdev. The vdev only grows after you have replaced every member with a larger disk. Then set autoexpand=on, or run zpool online -e tank DEVICE on each member. After that, zpool list shows the new size.