df Full, du No Agree: Find Where Space Dey Go

Why df full dey talk while du no dey agree

df dey report say disk full, but du no fit find the space because one process still dey hold a file wey dem don delete. When you delete file, e remove the file name from directory. The data blocks no go release until the last open file descriptor wey dey point to that inode close. du dey walk through file names, so e no count anything. df dey ask filesystem how many blocks dem allocate, so e still count the file wey no longer get name.

This guide go reproduce the issue for plain Ubuntu VPS with tools wey already dey installed. E go find the process wey dey hold the file through /proc, then free the space without reboot. Other causes of the same symptom follow: inode table wey no get free entries, files wey dey under mount point, and blocks wey dem reserve for root.

Run each command and read your own output. The values depend on your disk. So compare the before and after result for your own machine, instead of comparing am with figure wey guide print.

What df count and what du count

df (disk free) dey ask each mounted filesystem for its own accounting: how many blocks dey exist, how many dem allocate, and how many remain free. E no ever open directory. The answer cover every allocated block, including blocks wey belong to file wey no directory entry dey point to.

du (disk usage) dey do the opposite. E start for path wey you give am, read directories, check every entry wey e find, then add the blocks together. File wey no get name no dey visible to am. Any directory wey e no get permission to read no dey visible too. Na why ordinary user total dey smaller than root own. Run du under sudo before you conclude anything from the comparison.

Two options matter every time you compare the two.

• -x keep du for one filesystem. Without am, du / go enter every filesystem mounted below / and produce total wey df / no dey measure.
• -s print one summary line for each argument instead of one line for each directory.

This one give you the pair to run side by side for the filesystem wey you care about.

df -h /
sudo du -xhs / 2>/dev/null

df answer immediately. du fit take minutes for large filesystem, because e go check every file as e dey go. When the two totals far apart, and du run as root with -x, the missing space dey allocated to something wey no get name.

Reproduce the mismatch on purpose

Do this for one test VPS. Everything wey dey below na bash and coreutils, so nothing need install.

Record the starting state of the filesystem wey dey hold /var/tmp.

cd /var/tmp
df -h .
df --output=used -B1 .

The second command dey print used bytes without rounding, so the check for the end go exact.

Now create one file. The file size dey come from the free space wey the machine report by itself, so the demonstration go fit any disk wey you get.

free=$(df --output=avail -B1 . | tail -n 1)
fallocate -l $((free / 10)) ghost.bin
ls -l ghost.bin
df -h .

$(...) na command substitution: the shell dey run the command inside am, and the output go become the value of free. If this syntax new to you, command substitution for bash explain am properly. fallocate dey reserve real blocks without writing to dem, na why e dey finish immediately. For filesystem wey no support am, the command go fail, and head -c $((free / 10)) /dev/zero > ghost.bin go do the same work by writing the bytes out.

Compare this df -h . with the one wey you record. The used column don grow and the available column don reduce.

Now make another process hold the file open, then delete am.

sleep infinity < ghost.bin &
holder=$!
rm ghost.bin
ls -l ghost.bin
df -h .
sudo du -xhs . 2>/dev/null

The redirection na the whole trick. sleep infinity < ghost.bin & dey start one background process wey standard input na that file, so the shell open the file and hand the descriptor to sleep, wey keep am open. $! dey hold the process ID of that background job. rm then remove the name while the descriptor still dey open.

Read the output. ls no fit find the file because the name don disappear. du don return near where e start because e dey walk through names. df no move because the blocks still dey allocated. The filesystem and the directory tree no longer agree, and the gap between dem na the file wey you just delete.

Find di process wey dey hold deleted file

Every open file descriptor dey appear under /proc/<pid>/fd/ as symbolic link go the file wey e refer to. When dem don unlink the file, kernel dey mark the target of that link as deleted. So, to find wetin dey hold am, find link wey target get that marker.

sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

-lname dey match the target of symbolic link instead of the link name, %p dey print descriptor path, and %l dey print wetin the link point to. Process ID na the second part of the path wey e print. Run am with sudo, because without that you fit only read /proc/<pid>/fd for your own processes. The stderr redirect dey remove noise from processes wey exit while find dey scan.

Busy server fit dey hold several deleted files at any time, and most of dem small and harmless. Sort dem by size so only the important ones go remain for the top.

sudo bash -c 'for fd in /proc/[0-9]*/fd/*; do
  target=$(readlink "$fd" 2>/dev/null) || continue
  case "$target" in
    *"(deleted)") echo "$(stat -Lc %s "$fd" 2>/dev/null) $fd $target" ;;
  esac
done' | sort -rn | head

stat -L dey follow the link go the inode itself, so %s dey report the size of the file wey no longer get name. Sorting with that number go put the biggest one first.

Then identify the process behind the descriptor wey win. The path for the top of that list get both numbers wey you need, so put dem inside variables first. Replace PID and N with wetin your own listing print.

pid=PID
n=N
ps -o pid,user,etime,args -p "$pid"
sudo stat -L "/proc/$pid/fd/$n"

ps dey name the program and show how long e don dey run. stat -L dey print the size and allocated block count of the deleted inode. Together, dem answer the main question: which service dey keep this file alive.

If the machine already get lsof, sudo lsof +L1 dey list open files wey link count don drop to zero and show the sizes for one table. E no dey present for minimal Ubuntu image, and installing package for filesystem wey no get free space fit fail by itself. So, /proc walk na the version wey always work.

Free up the space without reboot

Reboot dey fix am, but e no be the correct first step: e go bring the service down and destroy the evidence. Four gentler options dey, and you suppose try dem in this order.

First, copy the data out if you still need am. If you read the descriptor path, you dey read the live inode.

sudo cp /proc/<pid>/fd/<n> /root/recovered.log

Na this one case deleted file easy to get back, na why recover files wey rm -rf delete dey start by asking whether process still dey hold the file open. Once the last descriptor close, you no fit use that route again.

Second, empty the file through the descriptor. The /proc path dey point to the same inode, so truncating am go release the blocks while the process continue to run.

sudo truncate -s 0 "/proc/$pid/fd/$n"
df -h /

This one dey work well when the writer open the file in append mode, because every write go enter the current end of the file. If e no be append mode, the process go keep the old write offset, so the next write go land far inside the file and recreate am with a hole for the front. Hole no dey allocate storage, so the blocks remain free and df keep the space wey e just return. Na only the size go come back: run sudo stat -L "/proc/$pid/fd/$n" again after the process write, and e go report the old size beside a block count wey no match am again. Restart the process when you want the size to start from zero too.

Third, ask the service to reopen its logs. Daemon wey delete its log file underneath am na the common real form of this problem. Plenty daemons dey reopen their log files when dem receive signal: nginx uses SIGUSR1 and rsyslog uses SIGHUP. Check the documentation for the daemon wey you dey work with instead of guessing, because wrong signal sent to wrong daemon fit stop am.

sudo systemctl kill -s USR1 nginx

That one go send the signal to whichever process systemd record as the unit main process. So, if unit declare the wrong Type= for how its daemon actually starts, your signal fit reach process wey never hold the deleted file, and the space go remain occupied.

Fourth, restart the unit. sudo systemctl restart <unit> go close every descriptor wey the old process hold, so the blocks go return for sure. For the demonstration above, the holder na sleep wey you start yourself, so to end am don dey enough.

kill $holder
df -h .
df --output=used -B1 .
sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

Compare the used bytes with the value wey you record before you create the file. Dem don agree again, and find no longer report your descriptor. Verify with the same command wey find the problem na the habit wey worth keeping.

Watching that value change easier than running df by hand again and again. watch dey repeat command for fixed interval and reprint the output for the same place, so watch df -h / go show the used column changing as the space dey come back.

Wetin to do when the totals agree but disk still full

If df and a root du -x agree with each other, no deleted file dey involved. The remaining causes dey different, and each one get its own check.

Inode don finish, but blocks still dey

Inode dey hold metadata for one file. ext4 dey create fixed number of inodes when dem make the filesystem, so filesystem fit run out of inodes even when free blocks still dey. New files go fail even though df -h show say space still dey.

df -h /
df -i /

The first command dey count blocks, and the second one dey count inodes. Compare the use column for both. If block use low but inode use don reach limit, the problem na very large number of very small files.

df no accept -i and --output for the same invocation. So when you want raw counts wey you fit read or pass to another command, select the inode fields by name and leave -i out.

df --output=itotal,iused,iavail,ipcent /

Those columns carry the same accounting wey df -i print, but for format wey you fit break down.

Find the files by counting entries instead of bytes.

sudo du --inodes -x -d 1 / 2>/dev/null | sort -rn | head

Run the same command again one level inside the directory wey come out on top. Continue until you reach the tree wey dey create the files. If your du no support --inodes, then sudo find /var -xdev -type f | wc -l dey count the subtree the slow way.

The solution na to delete or move those files. You no fit add inodes to an existing ext4 filesystem, because the count dey fixed when mkfs happen. To raise the count, you need recreate the filesystem and restore from backup. XFS dey allocate inodes as e need dem, so e no face fixed ceiling for the same way. Machine wey dey run containers go reach both limits faster than most machines, because image layers dey hold plenty small files. For that machine, pruning Docker disk usage for VPS na the specific solution, and e go reclaim much more space than general filesystem sweep.

Space wey hide under a mount point

Directory fit get files before anything mount on top of am. Mount filesystem over that directory, and the files underneath remain exactly where dem dey: dem still occupy space, still count for df, but you no fit reach dem by name again. du no fit see dem because the mount cover dem.

Use tmpfs to show am. tmpfs no need spare disk. You need machine wey you get permission to mount on for this part, so e go work for KVM VPS.

sudo mkdir -p /srv/covered
sudo cp /etc/services /srv/covered/
ls /srv/covered
sudo mount -t tmpfs tmpfs /srv/covered
ls /srv/covered
sudo umount /srv/covered
ls /srv/covered

The middle ls show empty directory. The copy no go anywhere: e still dey for root filesystem, and e go show again immediately you unmount. Now imagine say service dey write logs to that path for one month before person mount volume over am.

To find the real files for running server, mount root filesystem one more time for another location. Bind mount go show one filesystem without the filesystems wey mount inside am.

sudo mkdir -p /mnt/rootcheck
sudo mount --bind / /mnt/rootcheck
sudo du -xhs /mnt/rootcheck/* 2>/dev/null | sort -h
sudo umount /mnt/rootcheck

Anything wey appear for that listing but no dey under the normal path dey buried under a mount point. Unmount the bind mount when you finish, or a later du without -x go count the same files two times.

Blocks wey dem reserve for root

ext4 dey keep some blocks for root user, so full disk no go stop root from logging in and repairing the machine. Process wey dey run as ordinary user go hit that limit first, while df still go show small space. Read the setting for your own filesystem instead of assuming the default.

dev=$(df --output=source / | tail -n 1)
sudo tune2fs -l "$dev" | grep -i 'block count'

That one go print total block count and reserved block count with the same units, so you fit compare their ratio directly. df dey report the available column as space wey normal user fit still use. Na why used plus available dey smaller than the total size. The difference na the reserved space.

Use sudo tune2fs -m <percent> "$dev" to change am. The change dey apply immediately, and e no need remount. You fit reasonably lower the reserve for separate data filesystem. For root filesystem, leave enough space so root fit still write, because root filesystem wey no get any free space at all dey much harder to repair. This reserve also dey help prevent lockout: if you append a key to authorized_keys for filesystem wey no get space again, the key fit write incomplete or no write at all. Then the next login go answer with Permission denied (publickey) for reason wey no concern the key itself. tune2fs works with ext2, ext3 and ext4. XFS no get equivalent setting.

Where du dey mislead you by itself

Four habits of du dey produce totals wey look wrong.

• Hard links: du dey count one inode once even when plenty names point to am, so a tree full of hard links go report less than the sum of its files.
• Sparse files: du dey report the blocks wey system actually allocate, while ls -l dey report the apparent size. Add --apparent-size to see the other number.
• Permissions: if you run as ordinary user, du go skip wetin e no fit read and report less than the real value. The errors wey e print na the ones people redirect to /dev/null and stop reading.
• Filesystem boundaries: without -x, du / dey count every filesystem mounted below /, so its total fit pass wetin df / dey report.

df get one habit wey you suppose know too. E dey report each filesystem separately, so run am against the exact path wey the failed write dey target. A separate /boot dey fill according to its own schedule as kernel packages accumulate, and removing old kernels on Ubuntu na different job from clearing space on /.

Work order wey fit work for real incident

1. Run df -h <path> and df -i <path> for the filesystem wey failed write dey target, no be / by reflex.
2. Run sudo du -xh -d 1 <mountpoint> 2>/dev/null | sort -h, then move go inside the directory wey big pass.
3. If du no fit explain wetin df report as used, search /proc for deleted files wey still dey open.
4. If both agree, bind mount the filesystem for another location, then look for files wey dey under a mount point.
5. If inode use na wetin reach the limit, count files instead of bytes.

Every step for there get command wey output you fit read. Na this one make the difference between fixing the problem and just guessing.

FAQ

Why df dey show say disk full when du find far less?

The common reason na file wey dem delete while process still dey keep am open. When dem remove the file, dem remove the directory entry too, so du no get name to walk and e stop counting am. The inode and the blocks wey belong to am remain allocated until the last descriptor close, while df dey count allocated blocks. Search /proc/<pid>/fd for symbolic links wey target dey marked as deleted, and you go see both the file and the process wey dey hold am. Before you trust the comparison, confirm say you run du as root and with -x, because ordinary user go silently skip directories wey e no fit read.

How I fit find deleted file wey still dey open without lsof?

Use the kernel own record of open descriptors. sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null lists every descriptor wey point to file wey no get name, and process ID dey inside the path wey e print. sudo stat -Lc %s for one of those descriptor paths go report the size, so you fit sort dem and choose the one wey matter. You no need any package for this, and this matter because installing package for filesystem wey no get free space fit fail.

I fit free the space without killing the process?

Sometimes. sudo truncate -s 0 /proc/<pid>/fd/<n> reaches the same inode through the descriptor and releases its blocks while the process continue to run. This na the cleanest option when process open the file in append mode, because all its writes go to the current end. If e no do so, the write offset remain where e dey, and the next write go recreate the file with a hole for the front. The reported size go come back, while the blocks under the hole remain free. Restarting the unit, or sending the signal wey the documentation name so e go reopen its logs, na the fix wey no leave sparse file behind.

df dey show free space but writes still dey fail. Wetin else fit cause am?

Check inodes with df -i for the same path, because filesystem wey get free blocks but no free inodes go reject new files. Check whether non-root user dey run the write against ext4 filesystem where na reserved blocks remain. sudo tune2fs -l for the device go show this. Check say you dey read the filesystem wey the write really target, because separate /boot or /var fit fill independently of /.

Why du dey report total wey bigger pass df?

du without -x crosses into every filesystem mounted under the path wey you give am. So e adds several filesystems together, while df dey describe only one. Bind mounts make the matter worse, because the same files go count once under every path where dem appear. Add -x to keep du on one filesystem, and give df the same path, so both commands dey describe the same thing.