0

Yesterday evening, I ran something that produced way more output data that I expected, filling the root ext4 partition as a result. I killed the offending process and started deleting unrelated data I no longer needed, and then saw this:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.4G     0  3.4G   0% /dev
tmpfs           690M   11M  680M   2% /run
/dev/sda4       810G  806G     0 100% /
tmpfs           3.4G     0  3.4G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
/dev/sda1       476M   26M  450M   6% /boot/efi
tmpfs           690M     0  690M   0% /run/user/1000

As you can see, there is no space available even though "used" is four gigabytes smaller than "size". Working under the assumption that these are deleted files that some process has open, I used lsof to find that the biggest one was a 64MB /memfd:pulseaudio.

Baffled, I rebooted the system, being almost sure it would solve the issue. However, the output of df didn't change.

How is this possible and how can I fix it?

$ uname -a
Linux ... 4.16.0-1-amd64 #1 SMP Debian 4.16.5-1 (2018-04-29) x86_64 GNU/Linux
$ cat /etc/issue
Debian GNU/Linux buster/sid \n \l
Maya
  • 378

1 Answers1

0

ext4 has a concept of 'reserved blocks' that can only be filled by a process running as root. It might be that the output from df takes that into account.

For example, on one system, df shows:

Filesystem          1K-blocks      Used Available Use% Mounted on
/dev/sda1            30626752  14557916  14490036  51% /

Here, 30626752 - 14557916 - 14490036 equals 1578800, which is about 5 % of the total, the default amount for reserved blocks. Changing the reserved blocks percentage to zero with tune2fs (tune2fs -m 0 /dev/sda1) changes the numbers shown by df: 

Filesystem          1K-blocks      Used Available Use% Mounted on
/dev/sda1            30626752  14557924  16052444  48% /

(The figures still don't match exactly, there's 16384 1k blocks unaccounted for, but I assume that's due to some internal filesystem structure that's not counted as "used" for some reason.)

ilkkachu
  • 138,973