ZFS on FreeBSD and Linux: the RAM trade-off

What ZFS gives you, and what it takes

ZFS on FreeBSD and Linux is one codebase now, OpenZFS, so the features are the same on both systems. A server running ZFS gets checksummed data, snapshots that cost nothing until data changes, replication with zfs send, and compression that is one property away. What it takes is memory: the ARC (adaptive replacement cache) claims a large share of RAM by default, and on a 2 GB or 4 GB VPS (virtual private server) that memory is exactly what your application wanted.

This guide judges ZFS from a rented VPS with one or two virtual disks, not from a storage box with forty drive bays. The features that survive that move are the ones worth your time. The parts that do not survive it are worth knowing before you build a pool.

OpenZFS on FreeBSD and Linux: one codebase, two packaging stories

FreeBSD has carried ZFS in the base system since FreeBSD 7.0 in 2008, at first as an experimental feature. Since OpenZFS 2.0 in December 2020, FreeBSD and Linux build from the same source tree, so zfs and zpool behave the same way on both, and a pool created on one imports on the other.

The reason ZFS is a package on Linux and part of the base system on FreeBSD is licensing. OpenZFS is under the CDDL (common development and distribution license). The Linux kernel is under the GPL (general public license) version 2. The kernel project treats the two as incompatible, so the ZFS code is not merged into mainline Linux, and each distribution decides how to ship it. FreeBSD has no such conflict, so ZFS is simply there. That is the whole practical story: one packaging difference, and nothing you need to take a side on.

SSD Nodes does not offer FreeBSD images, so on a server rented here the Linux half of this guide is what applies. If you run FreeBSD somewhere else, a FreeBSD server gets ZFS with no module to build and no kernel upgrade to survive.

Install ZFS and create a pool

On Ubuntu the module ships inside the kernel packages, so you only install the commands.

sudo apt update
sudo apt install -y zfsutils-linux
zfs version

zfs version prints two lines, the userland version and the kernel module version. One line only means the module did not load. The package lives in the universe component, which Ubuntu server images enable by default; if apt cannot find it, run sudo add-apt-repository universe first.

On Debian the packages sit in the contrib component and the module is built on your machine by DKMS (dynamic kernel module support). Add contrib to the Components: line in /etc/apt/sources.list.d/debian.sources, run sudo apt update, then:

sudo apt install -y linux-headers-$(dpkg --print-architecture) zfs-dkms zfsutils-linux

The install compiles the module and prints Building initial module for 6.12.0-..., which takes a few minutes. Remember what that means: every kernel upgrade rebuilds it, and a build that fails leaves your pool unimported until you fix it.

On FreeBSD nothing is installed. Enable the service and start it.

sysrc zfs_enable=YES
service zfs start

Now the pool. Look at the stable device paths first, because /dev/vdb is handed out in detection order and can move when you attach another volume.

ls -l /dev/disk/by-id/
sudo zpool create -o ashift=12 tank /dev/disk/by-id/virtio-abc123def456
zpool status tank

zpool status should print state: ONLINE with your device listed under tank. ashift=12 fixes the pool's smallest block at 4 KiB, which matches current SSDs and cannot be changed after creation.

Most rented images boot from an ext4 root, so ZFS here is a data pool on a second volume rather than the root filesystem. Check the device is what you think it is before you build on it, because confirming the NVMe disk you were sold takes one minute and a rebuild takes an afternoon.

Checksums only repair when the pool has redundancy

Every block ZFS writes carries a checksum, and every read verifies it. Detection always works. Repair needs a second copy.

On a single-disk pool, ZFS tells you the truth and stops there. zpool status -v reports it like this:

status: One or more devices has experienced an error resulting in data
        corruption.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
errors: Permanent errors have been detected in the following files:

        /tank/data/archive.tar

The bad file is named. ext4 would have returned those bytes without comment, so this is already worth something. ZFS still cannot fix it, because there is no second copy in the pool to fix it from.

With a mirror the same read is served from the good side, the bad block is rewritten, and the event appears in the CKSUM column of zpool status. That is self-healing, and it needs two devices.

sudo zpool create -o ashift=12 tank mirror /dev/disk/by-id/DISK1 /dev/disk/by-id/DISK2

On a VPS the host storage is usually redundant already, often RAID 10 under the hypervisor. That protects you against a dead drive. It does not tell you when a block came back wrong, because the array has no way to know which copy is correct. ZFS knows, because it compares the data against a checksum it wrote itself.

If you have one virtual disk and want some repair ability, sudo zfs set copies=2 tank/important stores two copies of every block of that dataset on the same disk. It doubles the space that dataset uses, it survives a bad block, and it does nothing when the whole volume disappears.

A scrub reads everything in the pool and verifies it.

sudo zpool scrub tank
zpool status tank

A healthy pool ends with a line like scan: scrub repaired 0B in 00:04:11 with 0 errors. Put it on a schedule; monthly is fine for a small pool.

systemctl list-unit-files 'zfs-scrub*'
sudo systemctl enable --now zfs-scrub-monthly@tank.timer

Datasets are the unit of policy

A dataset is a filesystem inside the pool, and creating one is cheap, so make one per job. Properties inherit down from the pool, which means you set a default once and override it where it matters.

sudo zfs create tank/data
sudo zfs create tank/pg
sudo zfs set compression=lz4 tank
sudo zfs set atime=off tank
sudo zfs set quota=20G tank/data
sudo zfs set recordsize=16K tank/pg
zfs get -r compression,compressratio,quota tank

Compression is the property people leave off out of caution, and that is backwards. lz4 costs a small amount of CPU and reduces the bytes that have to reach the disk, so on compressible data it usually makes reads and writes faster. zstd compresses harder for more CPU, which suits logs and archives you rarely read back. Check what you are actually getting with zfs get compressratio tank, and remember the ratio only counts data written after the property was set.

recordsize is the largest block a dataset writes, 128K by default. A database writing 8 KiB pages into 128 KiB records turns one small write into a read of the whole record, a change, and a write back. Set recordsize=16K on the database dataset before you load the data, because the property applies to newly written blocks only.

quota is how you stop one dataset from filling the pool. A ZFS pool near 100% full gets slow and awkward to clean up, so leave headroom on purpose.

Snapshots cost nothing until data changes

ZFS never overwrites a live block. It writes a new block and updates the pointers, which is what copy-on-write means. A snapshot is a note saying "keep the blocks this dataset pointed at right now", so taking one is instant and free.

sudo zfs snapshot tank/data@2026-08-11
zfs list -t snapshot -o name,used,refer -r tank/data

The USED column for a snapshot is the space held only by that snapshot. It starts near zero and grows as you change or delete data, because the old blocks can no longer be released.

Getting a file back needs no restore step.

ls /tank/data/.zfs/snapshot/
cp /tank/data/.zfs/snapshot/2026-08-11/notes.txt /tank/data/notes.txt

The .zfs directory is hidden even from ls -a until you run sudo zfs set snapdir=visible tank/data.

Rollback throws away everything written since the snapshot.

sudo zfs rollback tank/data@2026-08-11

It refuses when newer snapshots exist, and -r destroys those newer snapshots to proceed. Read the dataset name twice before you press enter.

A snapshot is not a backup. It lives in the same pool, on the same volume, on the same server. A failed volume or one zpool destroy takes the snapshots along with the data. Snapshots protect you from your own rm and from a bad upgrade, which covers a lot of real incidents, and they protect you from nothing that happens to the pool itself. The full case is made here: why a VPS snapshot is not a backup.

Send and receive: replication in one command

zfs send turns a snapshot into a byte stream on standard output, and zfs receive turns that stream back into a dataset. The first copy is a full send.

sudo zfs snapshot tank/data@daily-2026-08-11
sudo zfs send tank/data@daily-2026-08-11 | ssh backup.example.com "sudo zfs recv -F backup/data"

After that, send only what changed between two snapshots.

sudo zfs snapshot tank/data@daily-2026-08-12
sudo zfs send -i tank/data@daily-2026-08-11 tank/data@daily-2026-08-12 | ssh backup.example.com "sudo zfs recv backup/data"

The receiving side must still hold the snapshot you are sending from. When it does not, the receive stops with cannot receive incremental stream: most recent snapshot of backup/data does not match incremental source, because ZFS has no base to apply the difference to. Send from a snapshot both sides hold, or start again with a full send.

Grant rights on the target instead of using remote root: sudo zfs allow -u backupuser create,mount,receive backup/data.

This is a real off-site backup with one condition. The far end must be a ZFS pool, since object storage cannot receive a stream. When your target is S3-compatible storage or a plain Linux host, use a tool that speaks to it, and restic backups from a VPS covers that path.

Why does ZFS use so much RAM? The ARC

The ARC (adaptive replacement cache) is ZFS's read cache. It sits in kernel memory rather than in the normal Linux page cache, so free -h does not report it under buff/cache. It reads as memory in use. A ZFS box that looks nearly full is usually a box with a warm cache, and that accounts for most "ZFS ate my RAM" reports.

The default limit is generous on purpose. OpenZFS 2.3 sets the maximum ARC size to the larger of RAM minus 1 GiB and 5/8 of RAM. OpenZFS 2.2 and earlier used half of RAM on Linux, while FreeBSD already used the newer rule. Run zfs version to see which one applies to you.

[
  {
    "label": "2 GB VPS",
    "openzfs_2_2_linux_gib": 1,
    "openzfs_2_3_gib": 1.25
  },
  {
    "label": "4 GB VPS",
    "openzfs_2_2_linux_gib": 2,
    "openzfs_2_3_gib": 3
  },
  {
    "label": "8 GB VPS",
    "openzfs_2_2_linux_gib": 4,
    "openzfs_2_3_gib": 7
  },
  {
    "label": "16 GB VPS",
    "openzfs_2_2_linux_gib": 8,
    "openzfs_2_3_gib": 15
  }
]

Those figures are the documented default rule applied to common instance sizes, not measurements from a running box. On a 4 GB instance the 2.3 rule allows an ARC of 3 GiB. The same box on 2.2 stops at 2 GiB. A 2 GB instance under the 2.3 rule still allows 1.25 GiB. Your application gets what is left.

Read the real numbers off your own server instead of trusting the table:

grep -E '^(size|c_max) ' /proc/spl/kstat/zfs/arcstats
arc_summary | head -n 20

The third column is bytes. c_max is the ceiling in force right now, and size is what the ARC currently holds.

The ARC does give memory back. The kernel signals pressure and the ARC shrinks. The problem is timing, because the shrink is driven by that pressure, so a process asking for several hundred MiB at once can meet the OOM (out of memory) killer while the ARC is still releasing. On a 2 GB box running a database and a web server, that is not a rare event. The OpenZFS manual says the same thing about manual changes: lowering the limit "will not cause the ARC to shrink without memory pressure to induce shrinking".

How to cap the ARC on a small VPS

Decide the workload's memory first. Add up what the database and the application need, keep a margin for the operating system, and give the rest to the ARC. On a 4 GB instance running Postgres and one web application, 512 MiB to 1 GiB of ARC is a sensible starting point.

Set it live, in bytes. This one is 1 GiB.

echo 1073741824 | sudo tee /sys/module/zfs/parameters/zfs_arc_max

Make it survive a reboot.

echo 'options zfs zfs_arc_max=1073741824' | sudo tee /etc/modprobe.d/zfs.conf
sudo update-initramfs -u

The initramfs step matters because the module can load from the initramfs before the root filesystem is mounted, which means it would never read the file you just wrote. After the reboot, confirm with the c_max line from arcstats.

Two caveats come from the manual itself. You cannot set the value back to 0 while the system runs, so undoing this means editing the file and rebooting. And lowering the number does not shrink a large ARC on the spot.

On FreeBSD the same limit is a sysctl under vfs.zfs.arc. Run sysctl vfs.zfs.arc to see the current values and the exact name your version uses, then write the maximum into /boot/loader.conf.

Two more memory rules for a small server. Leave deduplication off, because the dedup table lives in memory and the commonly published rule of thumb is 1 to 3 GB of RAM per TB of unique data. And do not put swap on a zvol (a block device carved out of the pool), because swapping through the filesystem that is trying to free memory can deadlock the machine. Keep swap on a plain partition or a swap file outside the pool.

When ext4 or XFS plus restic is the better answer

ZFS earns its keep on a server with spare memory and a second volume. Outside that, a plain filesystem plus a real backup tool wins. Choose ext4 or XFS when:

• The instance has 2 GB or 4 GB of RAM and the workload wants all of it.
• There is one virtual disk and no second copy, so ZFS gives you detection without repair.
• Your backup target is object storage or a plain Linux host, so nothing there can receive a zfs send stream.
• You run Debian with DKMS and cannot afford a kernel upgrade that leaves the module unbuilt.
• You need ZFS on the root filesystem and the provider's images only offer ext4.

Keep ZFS when you have a separate data volume, RAM to spare (8 GB and up is comfortable), and a plan that uses snapshots and zfs send rather than just enabling them. For everything else, ext4 with restic writing encrypted, deduplicated backups to storage the server does not control covers most of the same ground for none of the memory.

Failure modes, with the strings you will see

The pool is gone after a reboot. zpool status prints no pools available. The import service reads /etc/zfs/zpool.cache, so a pool missing from that file is never imported at boot. sudo zpool import lists what is importable, sudo zpool import tank brings it back, and sudo zpool set cachefile=/etc/zfs/zpool.cache tank makes it stick. A pool that was not exported cleanly from another system reports cannot import 'tank': pool may be in use from other system, and sudo zpool import -f tank overrides that once you are sure no other host has it.

modprobe: FATAL: Module zfs not found in directory /lib/modules/6.12.0-... on Debian after a kernel upgrade. DKMS did not build for the new kernel, usually because the matching headers are not installed. dkms status shows what is built for which kernel. sudo apt install -y linux-headers-$(uname -r) then sudo dkms autoinstall rebuilds it, and sudo zpool import tank brings the pool back.

The pool is full but you deleted the files. Deleted data stays on disk while a snapshot still references it, so du and df disagree. zfs list -o space -r tank splits usage into USEDDS and USEDSNAP, and a large USEDSNAP is your answer. Destroy the old snapshots with sudo zfs destroy tank/data@2026-06-01 and the space comes back.

CKSUM counts climbing in zpool status. Something below ZFS returned bad data. On a mirror the count is a warning and the block was repaired. On a single-disk pool the file is lost, zpool status -v names it, and you restore that one file from a backup that does not live in this pool.

The server is slow and swapping. Cap the ARC as above, then run arc_summary and look at the hit ratio. An ARC too small to hold the working set means every read goes to disk, which is the point at which a plain filesystem using the page cache would serve you better.

FAQ

How much RAM does ZFS need on a VPS?

ZFS runs on a 2 GB instance. The real question is what is left for your application. With no tuning, OpenZFS 2.3 lets the ARC grow to the larger of RAM minus 1 GiB and 5/8 of RAM, so a 4 GB box can hand 3 GiB to the cache. Set zfs_arc_max to a number your workload can spare, then confirm it by reading the c_max line from /proc/spl/kstat/zfs/arcstats.

Is a ZFS snapshot a backup?

No. A snapshot lives in the same pool as the data. It survives a bad rm and a failed upgrade, and it dies with the pool or the instance. Turn it into a backup by sending it to another machine with zfs send, or by running a backup tool that writes to storage this server does not control.

Does ZFS work the same on FreeBSD and Linux?

Same codebase since OpenZFS 2.0 in December 2020, same commands, same on-disk format, and pools move between them. The difference is packaging. FreeBSD ships ZFS in the base system. On Linux each distribution decides: Ubuntu builds the module into its kernel packages, while Debian builds it on your machine with DKMS, so a kernel upgrade can leave you without a module until the rebuild succeeds.

Can ZFS repair corruption on a VPS with one disk?

It detects the corruption and names the file, and it cannot repair it, because repair needs a second copy of the block. zfs set copies=2 on a dataset gives you that second copy at double the space, which handles a bad block but not a lost volume. A mirror across two volumes is the answer that actually heals.

Does compression slow the server down?

lz4 usually makes it faster. Compressed blocks mean fewer bytes written and fewer bytes read, and the CPU cost per block is small next to the disk it saves. Set compression=lz4 at the pool root so every dataset inherits it, then check zfs get compressratio tank once real data has been written.