0

Running Debian 12. According to df, I have only 22G remaining space on my main disk (nvme0n1p2); according to ncdu, I should have like 960G:

$ df -H
Filesystem                        Size  Used Avail Use% Mounted on
udev                               17G     0   17G   0% /dev
tmpfs                             3.4G  4.0M  3.4G   1% /run
/dev/nvme0n1p2                    1.1T  934G   22G  98% /
tmpfs                              17G     0   17G   0% /dev/shm
tmpfs                             5.3M  8.2k  5.3M   1% /run/lock
/dev/nvme0n1p1                    536M  6.2M  530M   2% /boot/efi
tmpfs                             3.4G   87k  3.4G   1% /run/user/1000
XXX.XXX.XXX.XXX:/volume1/data      27T   14T   14T  50% /mnt/data
XXX.XXX.XXX.XXX:/volume1/media     27T   14T   14T  50% /mnt/robmedia

# ncdu
.   8.2 TiB [######################] /mnt
   29.6 GiB [                      ] /var
    7.2 GiB [                      ] /usr
    3.6 GiB [                      ] /home
  224.8 MiB [                      ] /boot
  125.2 MiB [                      ] /opt
   12.0 MiB [                      ] /etc
.   3.8 MiB [                      ] /run
  136.0 KiB [                      ] /tmp
   44.0 KiB [                      ] /root
e  16.0 KiB [                      ] /lost+found
   12.0 KiB [                      ] /media
e   4.0 KiB [                      ] /srv
e   4.0 KiB [                      ] /.cache
.   0.0   B [                      ] /proc
    0.0   B [                      ] /sys
    0.0   B [                      ] /dev

Most of what I've found for this phantom drive problem suggests the culprit is usually an enormous reservation of space for a file that has been deleted. lsof +L1 lists many deleted files,maybe as many as a couple hundred, but they're small, and we're maybe talking about 5 GB total phantom space, not 960.

The answer may have something to do with drive overlays - # df -H shows me 20 overlays, one for each docker container I'm running. But when I stop the docker compose file and kill all those containers, the overlays go away but the problem remains.

What do?

SleepUncertainty
  • 11
  • 1
    like this, the ncdu output is meaningless for your problem, because it counts the data on other filesystems in your /. If anything, run ncdu -x /, and replace your ncdu output with the output of that :) afterwards, your /mnt should count 0! That is, unless you have files in there that do not actually reside on a different file system but your nvme0n1p2 – Marcus Müller Jan 07 '24 at 14:47
  • in any case, you'd clearly want to navigate in ncdu's UI into the /mnt directory and look what's happening there. – Marcus Müller Jan 07 '24 at 14:53
  • This looks disconcertingly similar to What's using all the space on a seemingly empty partition? – Chris Davies Jan 07 '24 at 17:07
  • @MarcusMüller I'm not worried about /mnt, which is connected to 2 folders on an NAS via nfs. I am beginning to wonder whether the main drive - nvme0n1p2 - is clogged with 900GB of metadata on the 14TB of videos in those two folders, but (a) I have no idea whether it's realistic for the metadata to be that big; (b) I feel like it shouldn't be that big because that would make NAS's unworkable; and (c) no idea how to fix it if so. – SleepUncertainty Jan 07 '24 at 17:28
  • In any case, though, # ncdu -x / does take /mnt down to zero, but my problem remains, which is that ncdu accounts for about 40 GB on a 1 TB drive that df thinks is 98% full. – SleepUncertainty Jan 07 '24 at 17:30
  • @SleepUncertainty it is not "connected to two folders"; there's a lot data in /mnt/, but not on these external volumes, as said twice now, – Marcus Müller Jan 07 '24 at 19:17
  • Yes, I understand you now, though the -x / option didn't reveal the files hiding in /mnt. Appreciate the help. – SleepUncertainty Jan 08 '24 at 20:17

1 Answers1

1

Figured it out - the bloating files were hiding in one of the /mnt directories on the main disk but were hidden from du because I had mounted the nfs drives on those points. Solution was to bind-mount / to /mnt/root so those files could be seen and disposed of without being masked. I also verified that unmounting the NAS folders did the same. Code:

mkdir /mnt/root
mount --bind / /mnt/root
ncdu /mnt/root

Apparently I used qbittorrent at some point when the NAS was not mounted and it took the opportunity to fill the mount point with nearly 1TB of video.

Credit to this Q&A:

du results on filesystem inconsistent with df

SleepUncertainty
  • 11