du not accounting for space shown by df

Question

I have a CentOS 7.2 (minimal) VM running in a vSphere cluster, which has a 16G drive formatted as XFS under the default full-disk usage partitioning scheme from the CentOS installer. The system is running a test httpd service (all software from the default centos repos), but its space usage on disk is growing. This is in part from problematic apache error/access logs (2+GB in size), and removing these longs frees up some space. What is eating the drive space isn't so much a concern for me (I'll just nuke and pave this eventually). However, when I run df an du to check file sizes, I get a discrepancy and this is what's confusing.

First I run df -h to view filesystem usage:

[root@svrhttp03 httpd]# df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/centos_svrhttp03-root   14G  4.8G  9.1G  35% /
devtmpfs                           902M     0  902M   0% /dev
tmpfs                              912M     0  912M   0% /dev/shm
tmpfs                              912M   97M  816M  11% /run
tmpfs                              912M     0  912M   0% /sys/fs/cgroup
/dev/sda1                          497M  169M  329M  34% /boot
tmpfs                              183M     0  183M   0% /run/user/0

This is showing 4.8G used on the root FS.

I then run the following to view the size of all items at the root of the filesystem:

[root@svrhttp03 httpd]# du -a -h -t 10K / 2> /dev/null | grep -v -E "[A-Z,a-z,0-9]/." | sort -h
16K /home
7.4M    /root
31M /etc
97M /run
143M    /boot
324M    /var
1.3G    /usr
1.9G    /

This shows only 1.9G in use, and the reported size of all directories add up to ~1.9G, as expected.

So how can I find that ~2.9G discrepancy between what df and du are claiming for data on disk?

Answer 1

This may not be a full answer, but I know of two sources for such discrepancies:

• df showing more space than du (a.k.a. impossibly huge files)

Sparse files can cause this. For example: /var/log/lastlog is reported as impossibly huge (like 1.2 TB) on many VMs due to a bug in VMWare. It's not really that big, it's just a sparse file. Handling is to ignore them. df is always more accurate than du for actual disk space free.

• df showing fuller disk than du indicates (a.k.a. invisible files eating disk space)

The usual cause for this is deleted files that are still held open by processes. This is often the case after an upgrade and before a reboot—all the old library files are still held open by the processes that have the filehandles, even though they are "deleted" and won't show up from du or ls.

The easy handling is to reboot, but you can handle it more surgically. For example, start with lsof | grep -c DEL and see how many deleted files are still open. (A certain amount is relatively normal, not necessarily pathological, but it's still the right place to look to understand the disk space discrepancy.)

Answer 2

While Wildcard's answer is absolutely correct maybe it'd make more sense to say why.

du sums the file size of all files recursively through directories from the path specified on the VFS which can include multiple mounted file systems.

df reports the amount of allocated and unallocated space of the file system.

Allocated space for a file isn't always the same as the files size. Most file systems allocate in blocks much larger than a byte therefor a file can in its worst case allocate an entire block and then only use one byte of it. So many small files will allocate much more space than the sum of their sizes. Further sparse files can have a larger file size than the space it allocates because it only allocates parts of the file that are non-zero data.