How to list all disks mounted on non-empty directories (shadowed directories)?

Question

My linux server reports high disk space usage for device /dev/sda4 mounted on / as shown below:

[root@stormtrooper03 /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       126G  114G  5.5G  96% /
tmpfs            32G     0   32G   0% /dev/shm
/dev/sda2       239M  118M  109M  53% /boot
/dev/sda1       150M  264K  150M   1% /boot/efi
/dev/sda5        63G   52M   60G   1% /home
/dev/sda6        63G   54M   60G   1% /tmp
/dev/sda7        63G  2.6G   58G   5% /usr
/dev/sda3       539G   11G  501G   3% /var
/dev/sdb1       917G  857G   51G  95% /data/1
/dev/sdc1       917G  861G   47G  95% /data/2
/dev/sdd1       917G  282G  627G  31% /data/3
/dev/sde1       917G  861G   47G  95% /data/4
/dev/sdf1       917G  858G   50G  95% /data/5

However, I know there is much less than this amount used by running du -sh /* as shown below. I need to clean files setting on this device but I can't find them!!

[root@stormtrooper03 /]# du -sh /*
6.7M    /bin
116M    /boot
3.7T    /data
356K    /dev
30M /etc
172K    /home
638M    /lib
20M /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt
3.8G    /opt
0   /proc
42M /root
9.2M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
2.1M    /tmp
2.5G    /usr
11G /var

Update I have found reason for this: a new disk was mounted on /data/3 which had 110GB of data and became hidden from the filesystem after mounting a disk on it.

The answer to the question why does mount happen over an existing directory explains how to uncover original data using mount --bind but it doesn't explain how to find if your system has such "shadowed" directories if you don't know ahead of time.

Is there a quick way to a list of disks mounted on non-empty directories without checking each one individually by mount --bind?

Answer 1

[EDIT]: Leaving this answer as a general note, it does not answer the refined version of the question.

KDirStat is a good graphical utility for seeing where large files and directories are on a system. It gives a heatmap and a sorted list.

Answer 2

I figured out the problem! By tracing actions on this server that are different from other servers I found that at one point the disk mounted on /data/3 stopped working so it was commented out in /etc/fstab. However processes in the system didn't know about this and continued to write to /data/3. which at this time is a directory in /. Then later the bad disk was replaced and the fstab entry was updated so now /data/3 is mounted on the new disk. Somehow the old data was hidden as it seems there is no pointer to it or something.

Once I knew this, I unmounted the disk /data/3 and now I was able to see 110GB of data setting there which I deleted and then remounted the disk and things are back to normal.

I don't know if there is any tool to find such orphan directory so to speak.

Answer 3

If you mount the file system that might have the hidden data elsewhere, then you can go within the location to the relative spot where mounts might hide the data and see if there any any data there.

If sudo ls -A $MOUNTPOINT produces any results, then there is something there at $MOUNTPOINT.

To see what's mounted run the mount command by itself.

Here's some code that you can use to look for hidden data.

#!/bin/bash

TMP="/media/root" # temp location to mount the directory that might have data hidden by a mount point
sudo mkdir -p "$TMP"
sudo mount --bind --read-only -- "/" "$TMP" # if "/" holds the mount point with the hidden by mountpoint data, replace "/" if needed.

# replace `$(mount | awk '{print $3}')` below with the directories you wish to test for hidden data
for LOCATION in $(mount | awk '{print $3}') ; do
    if [ -n "$(sudo ls -A "$TMP$LOCATION" 2>/dev/null)" ]; then 
        echo "data on unmounted \"$LOCATION\"";
    fi; 
done;
sudo umount "$TMP"

sample output:

data on unmounted "/run"
data on unmounted "/"
data on unmounted "/var"