df Full, du No See Space: How You Go Find Am

Why df dey say full but du dey show different thing

df dey report say disk full, while du no fit find the space because one process still dey hold file wey dem don delete. When you delete file, e remove the file name from directory. Data blocks go release only when the last open file descriptor wey dey point to that inode close. du dey walk through file names, so e no count anything. df dey ask filesystem how many blocks dem allocate, so e still count the file wey no longer get name.

This guide dey reproduce the issue for plain Ubuntu VPS with tools wey already install, find the process wey dey hold the file through /proc, and free the space without reboot. Other causes of the same symptom follow: inode table wey no get free entries, files wey dey hide under mount point, and blocks wey dem reserve for root.

Run each command and read your own output. The values depend on your disk, so compare the result before and after for your own machine instead of comparing am with figure wey guide print.

Wetin df count and wetin du count

df (disk free) dey ask each mounted filesystem for im own accounting: how many blocks dey exist, how many dem don allocate, and how many remain free. E no ever open directory. The answer cover every allocated block, including blocks wey belong to file wey no directory entry point to.

du (disk usage) dey do the opposite. E start for path wey you give am, read directories, check every entry wey e find, then add the blocks together. File wey no get name no dey visible to am. Any directory wey e no get permission to read no dey visible too. Na why ordinary user go get smaller total than root. Run du under sudo before you conclude anything from the comparison.

Two options matter every time you compare the two.

• -x keep du for one filesystem. Without am, du / go enter every filesystem mounted under / and produce total wey df / never dey measure.
• -s print one summary line for each argument instead of one line for each directory.

This one give you the pair to run side by side for the filesystem wey you care about.

df -h /
sudo du -xhs / 2>/dev/null

df answer immediately. du fit take minutes for large filesystem, because e go check every file as e dey move through. When the two totals far apart, and du run as root with -x, the missing space dey allocated to something wey no get name.

Reproduce the mismatch on purpose

Do am for test VPS. Everything wey dey below na bash and coreutils, so nothing go install.

Record the starting state of the filesystem wey dey hold /var/tmp.

cd /var/tmp
df -h .
df --output=used -B1 .

The second command dey print used bytes without rounding, so the check for the end go exact.

Now create file. Its size dey come from the free space wey the machine report by itself, so the demonstration go fit any disk wey you get.

free=$(df --output=avail -B1 . | tail -n 1)
fallocate -l $((free / 10)) ghost.bin
ls -l ghost.bin
df -h .

$(...) na command substitution: shell dey run the command inside am, and the output go become the value of free. If this syntax new to you, command substitution for bash explain am properly. fallocate dey reserve real blocks without writing dem, na why e dey finish instantly. For filesystem wey no support am, the command go fail, and head -c $((free / 10)) /dev/zero > ghost.bin dey do the same work by writing the bytes out.

Compare this df -h . with the one wey you record. The used column don grow, and the available column don reduce.

Now make another process hold the file open, then delete am.

sleep infinity < ghost.bin &
holder=$!
rm ghost.bin
ls -l ghost.bin
df -h .
sudo du -xhs . 2>/dev/null

The redirection na the whole trick. sleep infinity < ghost.bin & dey start background process wey standard input be that file, so shell open the file and hand the descriptor to sleep, wey keep am open. $! hold the process ID of that background job. rm then remove the name while the descriptor still dey open.

Read the output. ls no fit find the file, because the name don disappear. du don return near where e start, because e dey walk through names. df no move, because the blocks still dey allocated. The filesystem and the directory tree don now disagree, and the gap between dem na the file wey you just delete.

Find process wey dey hold deleted file

Every open file descriptor dey appear under /proc/<pid>/fd/ as symbolic link to the file wey e refers to. When dem unlink the file, kernel mark the target of that link as deleted. So, to find the holder, find link wey its target get that marker.

sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

-lname match the target of symbolic link instead of the link name, %p print the descriptor path, and %l print wetin e point to. Process ID na the second part of the path wey e print. Run am with sudo, because without that, you fit only read /proc/<pid>/fd for your own processes. The stderr redirect remove noise from processes wey exit while find dey walk through the entries.

Busy server fit hold several deleted files at any time, and most of dem small and harmless. Sort dem by size so only the important ones stay for top.

sudo bash -c 'for fd in /proc/[0-9]*/fd/*; do
  target=$(readlink "$fd" 2>/dev/null) || continue
  case "$target" in
    *"(deleted)") echo "$(stat -Lc %s "$fd" 2>/dev/null) $fd $target" ;;
  esac
done' | sort -rn | head

stat -L follow the link go the inode itself, so %s report the size of the file wey no longer get name. Sorting with that number put the biggest one first.

Then identify the process behind the descriptor wey win. The path for top of that list get both numbers wey you need, so put dem inside variables first. Replace PID and N with wetin your own listing print.

pid=PID
n=N
ps -o pid,user,etime,args -p "$pid"
sudo stat -L "/proc/$pid/fd/$n"

ps name the program and show how long e don dey run. stat -L print the size and allocated block count of the deleted inode. Together, dem answer the important question: which service dey keep this file alive.

If the machine already get lsof, sudo lsof +L1 list open files wey their link count don drop to zero and show the sizes for one table. E no dey present for minimal Ubuntu image, and installing package for filesystem wey no get free space fit fail by itself. So, the /proc walk na the version wey always work.

Free the space without a reboot

Reboot go fix am, but na wrong first move: e go bring the service down and e go destroy the evidence. Four gentler options dey, in the order wey you suppose try dem.

First, copy the data comot if you still want am. Reading the descriptor path go read the live inode.

sudo cp /proc/<pid>/fd/<n> /root/recovered.log

Na this one case wey deleted file easy to get back, na why recover files wey rm -rf delete dey start by asking whether process still hold the file open. Once the last descriptor close, that route don disappear.

Second, empty the file through the descriptor. The /proc path lead to the same inode, so truncating am go release the blocks while the process still dey run.

sudo truncate -s 0 "/proc/$pid/fd/$n"
df -h /

This one work well when the writer open the file in append mode, because every write go then go the current end of the file. When e no be so, the process go keep the old write offset, so the next write go land far inside the file and recreate am with a hole for front. Hole no dey get allocation, so the blocks remain free and df go keep the space wey e just return. Na only the size go come back: run sudo stat -L "/proc/$pid/fd/$n" again after the process write, and e go report the old size beside a block count wey no longer match am. Restart the process when you want the size to start from zero too.

Third, ask the service to reopen its logs. Daemon wey dem delete the log file from under am na the common real version of this problem. Plenty daemons dey reopen their log files when dem receive signal: nginx dey use SIGUSR1 and rsyslog dey use SIGHUP. Check the documentation for the daemon wey dey in front of you instead of guessing, because wrong signal sent to wrong daemon fit stop am.

sudo systemctl kill -s USR1 nginx

Fourth, restart the unit. sudo systemctl restart <unit> go close every descriptor wey the old process hold, so the blocks go come back for sure. For the demonstration above, the holder na sleep wey you start yourself, so ending am go dey enough.

kill $holder
df -h .
df --output=used -B1 .
sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

Compare the used bytes with the value wey you record before you create the file. Dem agree again, and find no longer report your descriptor. Verifying with the same command wey find the problem na the habit worth keeping.

Watching that value change easier pass running df by hand again and again. watch dey repeat command for fixed interval and reprint the output for the same place, so watch df -h / go show the used column change as the space dey come back.

When the totals agree and the disk still full

If df and a root du -x agree with each other, no deleted file dey involved. The remaining causes dey different, and each one get im own check.

Inode don finish, blocks never finish

Inode dey hold metadata for one file. ext4 dey create fixed number of inodes when dem make filesystem, so filesystem fit run out of inodes while free blocks still dey. New files go fail even though df -h show say space still dey.

df -h /
df -i /

The first command dey count blocks and the second one dey count inodes. Compare the use column for both. If block use low but inode use don reach limit, the problem na very large number of very small files.

df no dey accept -i and --output for the same invocation, so when you want make the raw counts easy to read or pass to another command, select the inode fields by name and leave -i off.

df --output=itotal,iused,iavail,ipcent /

Those columns get the same accounting information wey df -i dey print, but this format make you fit break am down.

Find the files by counting entries instead of bytes.

sudo du --inodes -x -d 1 / 2>/dev/null | sort -rn | head

Run the same command again one level down inside the directory wey get the highest count, until you reach the tree wey dey create the files. If your du no support --inodes, then sudo find /var -xdev -type f | wc -l go count the subtree the slow way.

The solution na to delete or move those files. You no fit add inodes to an existing ext4 filesystem, because the count don fixed for mkfs time. To increase the count, you must recreate the filesystem and restore from backup. XFS dey allocate inodes as e need dem, so e no face fixed ceiling the same way. Machine wey dey run containers go reach both limits earlier than most machines, because image layers dey hold plenty small files. For that machine, prune Docker disk usage for a VPS na the specific solution, and e go reclaim much more space than general filesystem sweep.

Espa wey dey hide under mount point

Directory fit get files before anything mount for top of am. Mount filesystem over that directory, and the files underneath remain exactly where dem dey: dem still dey allocated, still dey counted by df, but you no fit reach dem by name again. du no fit see dem because the mount cover dem.

Use tmpfs show am, because e no need spare disk. This part need machine wey you get permission to mount, so e go work for KVM VPS.

sudo mkdir -p /srv/covered
sudo cp /etc/services /srv/covered/
ls /srv/covered
sudo mount -t tmpfs tmpfs /srv/covered
ls /srv/covered
sudo umount /srv/covered
ls /srv/covered

The middle ls show empty directory. The copy no go anywhere: e still dey root filesystem, and e go return as soon as you unmount. Now imagine service wey log into that path for one month before person mount volume over am.

To find the real files for running server, mount root filesystem one more time for another location. Bind mount show one filesystem without the filesystems mounted inside am.

sudo mkdir -p /mnt/rootcheck
sudo mount --bind / /mnt/rootcheck
sudo du -xhs /mnt/rootcheck/* 2>/dev/null | sort -h
sudo umount /mnt/rootcheck

Anything wey appear for that listing but no dey under the normal path dey buried beneath mount point. Unmount the bind mount when you finish, or later du without -x go count the same files two times.

Blocks wey reserve for root

ext4 dey keep some blocks for root user, so full disk no go stop root from login and repair the machine. Process wey ordinary user dey run go hit this limit first, while df still dey show small space. Read the setting for your own filesystem instead of assuming the default.

dev=$(df --output=source / | tail -n 1)
sudo tune2fs -l "$dev" | grep -i 'block count'

That command dey print total block count and reserved block count with the same units, so you fit calculate the ratio directly. df dey report the available column as the space wey normal user fit still use. Na why used plus available dey smaller than the total size. The difference na the reserved space.

Use sudo tune2fs -m <percent> "$dev" to change am. The change dey apply immediately and e no need remount. You fit lower the reserve for separate data filesystem. For root filesystem, leave enough space so root fit still write, because root filesystem wey no get any free space at all dey much harder to repair. tune2fs works for ext2, ext3 and ext4. XFS no get equivalent setting.

Where du fit mislead you by itself

Four habits of du dey produce totals wey look wrong.

• Hard links: du counts an inode once even when plenty names point to am, so a tree full of hard links go report less than the total of its files.
• Sparse files: du reports the blocks wey e actually allocate, while ls -l reports the apparent size. Add --apparent-size to see the other number.
• Permissions: when you run am as ordinary user, du go skip wetin e no fit read and report less than the real value. The errors wey e print na the ones people redirect to /dev/null and stop reading.
• Filesystem boundaries: without -x, du / go count every filesystem mounted below /, so its total fit pass wetin df / reports.

df get one habit wey you suppose know too. E reports each filesystem separately, so run am against the exact path wey the failed write dey target. A separate /boot dey fill according to its own schedule as kernel packages dey accumulate, and removing old kernels for Ubuntu na different work from clearing space for /.

Order wey work for real incident

1. Run df -h <path> and df -i <path> for the filesystem wey failed write dey target, no be say make you run am for / by reflex.
2. Run sudo du -xh -d 1 <mountpoint> 2>/dev/null | sort -h, then move enter the directory wey big pass.
3. If du no fit explain wetin df report as used, search /proc for deleted files wey still dey open.
4. If both agree, bind mount the filesystem for another location, then look for files wey dey under a mount point.
5. If inode use na wetin reach the limit, count files instead of bytes.

Every step for there get command wey you fit read the output. Na this one separate correct fix from guesswork.

FAQ

Why df dey show say disk full when du find much less?

The usual reason na file wey dem delete while process still get am open. When dem remove the file, dem remove the directory entry, so du no get name to walk and e stop to count am. The inode and its blocks still dey allocated until the last descriptor close, while df dey count allocated blocks. Search /proc/<pid>/fd for symbolic links wey target dey marked as deleted, and you go see both the file and the process wey dey hold am. Before you trust the comparison, confirm say you run du as root and with -x, because ordinary user go silently skip directories wey e no fit read.

How I fit find deleted file wey still dey open without lsof?

Use the kernel own record of open descriptors. sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null lists every descriptor wey point to file wey no get name, and the process ID dey inside the path wey e print. sudo stat -Lc %s for one of those descriptor paths go report its size, so you fit sort dem and choose the one wey matter. You no need any package for this, and that matter because installing one for filesystem wey no get free space fit fail.

I fit free the space without killing the process?

Sometimes. sudo truncate -s 0 /proc/<pid>/fd/<n> reaches the same inode through the descriptor and releases its blocks while the process still dey run. This one clean pass when the process open the file in append mode, because all its writes dey go the current end. If e no do that, the write offset go remain where e dey, and the next write go recreate the file with a hole for the front. The reported size go come back, while the blocks under the hole still remain free. Restarting the unit, or sending signal wey the documentation name to make am reopen its logs, na the fix wey leave no sparse file behind.

df shows free space but writes still fail. Wetin else fit cause am?

Check inodes with df -i for the same path, because filesystem wey get free blocks but no free inodes go reject new files. Check whether non-root user dey run the write against ext4 filesystem where na only reserved blocks remain. sudo tune2fs -l for the device go show this. Check say you dey read the filesystem wey the write actually targets, because separate /boot or /var fit fill independently of /.

Why du dey report bigger total than df?

du without -x crosses into every filesystem mounted under the path wey you give am, so e adds several filesystems together while df dey describe only one. Bind mounts make am worse, because the same files go count once under every path wey dem appear. Add -x to keep du on one filesystem, and give df the same path, so both commands dey describe the same thing.