df says full, du says otherwise: find the space

Why df says full and du says otherwise

df reports the disk as full while du cannot find the space because a process still holds a file that was deleted. Deleting a file removes its name from a directory. The data blocks are released only when the last open file descriptor pointing at that inode is closed. du walks names, so it counts nothing. df asks the filesystem how many blocks are allocated, so it still counts the file that no longer has a name.

This guide reproduces that on a plain Ubuntu VPS with tools that are already installed, finds the holding process through /proc, and frees the space without a reboot. The other causes of the same symptom follow: an inode table with no free entries, files buried under a mount point, and blocks reserved for root.

Run each command and read your own output. The values depend on your disk, so compare the before and the after on your own machine instead of comparing to a figure printed in a guide.

What df counts and what du counts

df (disk free) asks each mounted filesystem for its own accounting: how many blocks exist, how many are allocated, how many are free. It never opens a directory. The answer covers every allocated block, including blocks belonging to a file that no directory entry points at.

du (disk usage) does the opposite. It starts at a path you give it, reads directories, stats every entry it finds, and adds up the blocks. A file with no name is invisible to it. So is any directory it is not allowed to read, which is why an ordinary user gets a smaller total than root does. Run du under sudo before you conclude anything from the comparison.

Two options matter every time you compare the two.

• -x keeps du on one filesystem. Without it, du / walks into every filesystem mounted below / and produces a total that df / was never measuring.
• -s prints one summary line per argument instead of one line per directory.

That gives the pair to run side by side on the filesystem you care about.

df -h /
sudo du -xhs / 2>/dev/null

df answers immediately. du takes minutes on a large filesystem, because it stats every file on the way. When the two totals are far apart, and du ran as root with -x, the missing space is allocated to something that has no name.

Reproduce the mismatch on purpose

Do this on a test VPS. Everything below is bash and coreutils, so nothing gets installed.

Record the starting state of the filesystem that holds /var/tmp.

cd /var/tmp
df -h .
df --output=used -B1 .

The second command prints used bytes with no rounding, which makes the check at the end exact.

Now create a file. Its size comes from the free space the machine itself reports, so the demonstration fits whatever disk you have.

free=$(df --output=avail -B1 . | tail -n 1)
fallocate -l $((free / 10)) ghost.bin
ls -l ghost.bin
df -h .

$(...) is command substitution: the shell runs the command inside it, and the output becomes the value of free. If that syntax is new to you, command substitution in bash covers it properly. fallocate reserves real blocks without writing them, which is why it finishes instantly. On a filesystem that does not support it the command fails, and head -c $((free / 10)) /dev/zero > ghost.bin does the same job by writing the bytes out.

Compare this df -h . with the one you recorded. The used column has grown and the available column has shrunk.

Now hold the file open from another process, then delete it.

sleep infinity < ghost.bin &
holder=$!
rm ghost.bin
ls -l ghost.bin
df -h .
sudo du -xhs . 2>/dev/null

The redirection is the whole trick. sleep infinity < ghost.bin & starts a background process whose standard input is that file, so the shell opens the file and hands the descriptor to sleep, which keeps it open. $! holds the process ID of that background job. rm then removes the name while the descriptor is still open.

Read the output. ls cannot find the file, because the name is gone. du is back near where it started, because it walks names. df has not moved, because the blocks are still allocated. The filesystem and the directory tree now disagree, and the gap between them is the file you just deleted.

Find the process holding the deleted file

Every open file descriptor appears under /proc/<pid>/fd/ as a symbolic link to the file it refers to. When the file has been unlinked, the kernel marks the target of that link as deleted. So finding the holder means finding a link whose target carries that marker.

sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

-lname matches the target of a symbolic link rather than its name, %p prints the descriptor path, and %l prints what it points at. The process ID is the second element of the path it prints. Run it with sudo, because otherwise you can only read /proc/<pid>/fd for your own processes. The stderr redirect drops the noise from processes that exit while find is walking.

A busy server holds several deleted files at any moment, and most of them are small and harmless. Sort them by size so only the interesting ones stay at the top.

sudo bash -c 'for fd in /proc/[0-9]*/fd/*; do
  target=$(readlink "$fd" 2>/dev/null) || continue
  case "$target" in
    *"(deleted)") echo "$(stat -Lc %s "$fd" 2>/dev/null) $fd $target" ;;
  esac
done' | sort -rn | head

stat -L follows the link to the inode itself, so %s reports the size of the file that no longer has a name. Sorting on that number puts the largest one first.

Then identify the process behind the winning descriptor. The path at the top of that list carries both numbers you need, so put them into variables first, replacing PID and N with what your own listing printed.

pid=PID
n=N
ps -o pid,user,etime,args -p "$pid"
sudo stat -L "/proc/$pid/fd/$n"

ps names the program and shows how long it has been running. stat -L prints the size and the allocated block count of the deleted inode. Together they answer the question that matters: which service is keeping this file alive.

If the machine already has lsof, sudo lsof +L1 lists open files whose link count has dropped to zero and shows the sizes in one table. It is not present on a minimal Ubuntu image, and installing a package on a filesystem with no free space can itself fail, so the /proc walk is the version that always works.

Free the space without a reboot

A reboot does fix it, and it is the wrong first move: it takes the service down and it destroys the evidence. Four gentler options exist, in the order to try them.

First, copy the data out if you still want it. Reading the descriptor path reads the live inode.

sudo cp /proc/<pid>/fd/<n> /root/recovered.log

This is the one case where a deleted file is easy to get back, which is why recovering files deleted with rm -rf starts by asking whether a process still holds the file open. Once the last descriptor closes, that route is gone.

Second, empty the file through the descriptor. The /proc path leads to the same inode, so truncating it releases the blocks while the process keeps running.

sudo truncate -s 0 "/proc/$pid/fd/$n"
df -h /

This works cleanly when the writer opened the file in append mode, because every write then goes to the current end of the file. When it did not, the process keeps its old write offset, so its next write lands far into the file and recreates it with a hole at the front. A hole is not allocated, so the blocks stay free and df keeps the space it just returned. What springs back is the size alone: run sudo stat -L "/proc/$pid/fd/$n" again after the process writes, and it reports the old size next to a block count that no longer matches it. Restart the process when you want the size to start from zero as well.

Third, ask the service to reopen its logs. A daemon whose log file was deleted underneath it is the common real version of this problem. Many daemons reopen their log files on a signal: nginx uses SIGUSR1 and rsyslog uses SIGHUP. Check the documentation for the daemon in front of you instead of guessing, because the wrong signal sent to the wrong daemon stops it.

sudo systemctl kill -s USR1 nginx

Fourth, restart the unit. sudo systemctl restart <unit> closes every descriptor the old process held, so the blocks come back with certainty. For the demonstration above the holder is a sleep you started yourself, so ending it is enough.

kill $holder
df -h .
df --output=used -B1 .
sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

Compare the used bytes against the value you recorded before you created the file. They agree again, and the find no longer reports your descriptor. Verifying with the same command that found the problem is the habit worth keeping.

Watching that value move is easier than running df by hand over and over. watch repeats a command on a fixed interval and reprints the output in place, so watch df -h / shows the used column change as the space comes back.

When the totals agree and the disk is still full

If df and a root du -x agree with each other, no deleted file is involved. The remaining causes are different in kind, and each has its own check.

Out of inodes, not out of blocks

An inode holds the metadata for one file. ext4 creates a fixed number of inodes when the filesystem is made, so a filesystem can run out of inodes while it still has free blocks. New files then fail even though df -h shows room.

df -h /
df -i /

The first command counts blocks and the second counts inodes. Compare the use column of each. Block use low and inode use at its limit means the problem is a very large number of very small files.

df rejects -i and --output in the same invocation, so when you want the raw counts to read or to hand to another command, select the inode fields by name and leave -i off.

df --output=itotal,iused,iavail,ipcent /

Those columns carry the same accounting df -i prints, in a form you can pick apart.

Find the files by counting entries instead of bytes.

sudo du --inodes -x -d 1 / 2>/dev/null | sort -rn | head

Repeat the same command one level down on the directory that came out on top, until you reach the tree that is creating the files. If your du does not support --inodes, then sudo find /var -xdev -type f | wc -l counts a subtree the slow way.

The cure is deleting or moving those files. You cannot add inodes to an existing ext4 filesystem, because the count is fixed at mkfs time, so raising it means recreating the filesystem and restoring from a backup. XFS allocates inodes as it needs them, so it does not meet a fixed ceiling in the same way. A machine that runs containers reaches both limits sooner than most, since image layers hold many small files. On that machine pruning Docker's disk usage on a VPS is the specific cure, and it reclaims far more than a general sweep of the filesystem will.

Space hidden under a mount point

A directory can hold files before anything is mounted on it. Mount a filesystem over that directory and the files underneath stay exactly where they were: still allocated, still counted by df, and no longer reachable by name. du cannot see them because the mount covers them.

Show it with tmpfs, which needs no spare disk. This part needs a machine where you are allowed to mount, so it works on a KVM VPS.

sudo mkdir -p /srv/covered
sudo cp /etc/services /srv/covered/
ls /srv/covered
sudo mount -t tmpfs tmpfs /srv/covered
ls /srv/covered
sudo umount /srv/covered
ls /srv/covered

The middle ls shows an empty directory. The copy never went anywhere: it is still on the root filesystem, and it returns the moment you unmount. Now picture a service that logged into that path for a month before someone mounted a volume over it.

To find the real thing on a running server, mount the root filesystem a second time somewhere else. A bind mount shows one filesystem without the filesystems mounted inside it.

sudo mkdir -p /mnt/rootcheck
sudo mount --bind / /mnt/rootcheck
sudo du -xhs /mnt/rootcheck/* 2>/dev/null | sort -h
sudo umount /mnt/rootcheck

Anything that appears in that listing but not under the normal path is buried beneath a mount point. Unmount the bind mount when you are finished, or a later du without -x counts the same files twice.

Blocks reserved for root

ext4 sets aside a share of its blocks for the root user, so that a full disk does not stop root from logging in and repairing the machine. A process running as an ordinary user hits that wall first, while df still shows a little room. Read the setting on your own filesystem instead of assuming the default.

dev=$(df --output=source / | tail -n 1)
sudo tune2fs -l "$dev" | grep -i 'block count'

That prints the total block count and the reserved block count in the same units, so the ratio between them is direct. df reports the available column as the space a normal user may still use, which is why used plus available comes out smaller than the size. The gap is the reserve.

Change it with sudo tune2fs -m <percent> "$dev". The change applies immediately and needs no remount. Lowering the reserve on a separate data filesystem is reasonable. On the root filesystem, leave enough that root can still write, because a root filesystem with nothing free at all is much harder to repair. tune2fs works on ext2, ext3 and ext4. XFS has no equivalent setting.

Where du misleads you on its own

Four habits of du produce totals that look wrong.

• Hard links: du counts an inode once even when several names point at it, so a tree full of hard links reports less than the sum of its files.
• Sparse files: du reports the blocks actually allocated, while ls -l reports the apparent size. Add --apparent-size to see the other number.
• Permissions: run as an ordinary user, du skips what it cannot read and under-reports. The errors it prints are the ones people redirect to /dev/null and stop reading.
• Filesystem boundaries: without -x, du / counts every filesystem mounted below /, so its total can exceed what df / reports.

df has one habit worth knowing as well. It reports each filesystem separately, so run it against the exact path the failing write targets. A separate /boot fills on its own schedule as kernel packages accumulate, and removing old kernels on Ubuntu is a different job from clearing space on /.

A working order for a real incident

1. Run df -h <path> and df -i <path> on the filesystem the failed write was targeting, not on / by reflex.
2. Run sudo du -xh -d 1 <mountpoint> 2>/dev/null | sort -h, then walk down into the largest directory.
3. If du cannot account for what df reports as used, search /proc for deleted files that are still open.
4. If the two agree, bind mount the filesystem elsewhere and look for files sitting under a mount point.
5. If inode use is what sits at the limit, count files rather than bytes.

Every step there is a command whose output you can read. That is the difference between fixing this and guessing at it.

FAQ

Why does df show the disk as full when du finds much less?

The usual reason is a file that was deleted while a process still had it open. Removing the file removes its directory entry, so du has no name to walk and stops counting it. The inode and its blocks stay allocated until the last descriptor closes, and df counts allocated blocks. Search /proc/<pid>/fd for symbolic links whose target is marked as deleted and you have both the file and the process holding it. Before trusting the comparison, confirm you ran du as root and with -x, because an ordinary user silently skips directories it cannot read.

How do I find a deleted file that is still open without lsof?

Use the kernel's own record of open descriptors. sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null lists every descriptor pointing at a file with no name, and the process ID sits inside the path it prints. sudo stat -Lc %s on one of those descriptor paths reports its size, so you can sort them and pick the one that matters. This needs no package at all, which matters because installing one on a filesystem with no free space can fail.

Can I free the space without killing the process?

Sometimes. sudo truncate -s 0 /proc/<pid>/fd/<n> reaches the same inode through the descriptor and releases its blocks while the process keeps running. That is cleanest when the process opened the file in append mode, because its writes always go to the current end. If it did not, the write offset stays where it was and the next write recreates the file with a hole at the front, so the reported size springs back while the blocks under the hole stay free. Restarting the unit, or signalling it to reopen its logs with the signal its documentation names, is the fix that leaves no sparse file behind.

df shows free space but writes still fail. What else can it be?

Check inodes with df -i on the same path, since a filesystem with free blocks and no free inodes rejects new files. Check whether the write runs as a non-root user against an ext4 filesystem where only the reserved blocks are left, which sudo tune2fs -l on the device will show. Check that you are reading the filesystem the write actually targets, because a separate /boot or /var fills independently of /.

Why does du report a bigger total than df?

du without -x crosses into every filesystem mounted under the path you gave it, so it adds up several filesystems while df describes one. Bind mounts make it worse, because the same files are counted once under each path they appear at. Add -x to keep du on a single filesystem, and give df the same path, so both commands are describing the same thing.