0

I'm using Ubuntu Linux 16. I'm running out of disk space ...

[rails@server ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/simfs       30G   29G  1.5G  96% /

but what I don't understand is when I get the total size of the "/" directory, it is showing way less than 29G ...

[rails@server ~]$ sudo du -sh /
du: cannot access ‘/proc/883/task/883/fd/4’: No such file or directory
du: cannot access ‘/proc/883/task/883/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/883/fd/4’: No such file or directory
du: cannot access ‘/proc/883/fdinfo/4’: No such file or directory
4.5G    /

What's going on here? More importantly, how do I figure out where all this mystery disk space is getting used?

Edit:

Output as requested by GracefulRestart ...

[rails@server ~]$ sudo lsof -nP 2>&1 | grep 'deleted'
postmaste  443      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  443      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  469      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  469      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  470      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  470      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  471      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  471      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  472      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  472      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
httpd     1130          root  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1130          root  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1133   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1133   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1134   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1134   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1135   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1135   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1136   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1136   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1137   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1137   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1138   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1138   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1139   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1139   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1140   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1140   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1141   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1141   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1142   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1142   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1143   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1143   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1144   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1144   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1145   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1145   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1146   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1146   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1147   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1147   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1148   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1148   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1149   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1149   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1132        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1132        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1158        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1158        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
postmaste 1270      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste 1270      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)

Edit 2: Here's the output from the second command ...

[rails@server ~]$ sudo du -h --max-depth=1 /; ls -plah / | grep -v /
214M    /opt
4.0K    /media
12K /boot
6.7M    /run
4.0K    /lost+found
du: cannot access ‘/proc/3478/task/3478/fd/4’: No such file or directory
du: cannot access ‘/proc/3478/task/3478/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/3478/fd/4’: No such file or directory
du: cannot access ‘/proc/3478/fdinfo/4’: No such file or directory
0   /proc
4.0K    /dev
4.0K    /.cpt_hardlink_dir_a920e4ddc233afddc9fb53d26c392319
114M    /tmp
0   /sys
14M /etc
3.6M    /root
396M    /home
12K /.rbenv
4.0K    /srv
1.9G    /usr
1.9G    /var
8.0K    /mnt
4.5G    /
total 64K
-rw-r--r--  1 root root    0 May  5  2018 .autorelabel
-rw-r--r--  1 root root    0 Nov  8 17:14 reboot
-rw-r--r--  1 root root    0 Nov  8 17:15 .vzfifo
Dave
  • 2,548
  • 1
    try with a bind mount https://unix.stackexchange.com/a/358331/30851 or check from live cd – frostschutz Nov 08 '19 at 21:53
  • That's a useful utilty, but sadly it only confirmed what I already new -- the top usage directory within "/" is "/var" at 1.9 GB. So still uncertain what the issue is. – Dave Nov 08 '19 at 22:09
  • It is possible for deleted files to hold disk space on a disk. See if you get any output for lsof -nP 2>&1 | grep 'deleted'. The column before the path should be a file size. – GracefulRestart Nov 08 '19 at 22:13
  • @GracefulRestart, that produced a whole bunch of stuff, which I included as an edit to my question. However, I'm a little clueless as to what it means and what to do next. For instance, is "(deleted)/dev/zero" a file path that I can delete? – Dave Nov 08 '19 at 22:34
  • According to that output, postmaster (PostgreSQL) and httpd (Apache) are the processes holding those deleted files open. Restarting those processes should release those holds. Unfortunately, it does not look like that would equal the ~25GB of missing space... What does the output of du -h --max-depth=1 /; ls -plah / | grep -v / show? Any large files at the root (/)? – GracefulRestart Nov 08 '19 at 23:03
  • @GracefulRestart, I included the output from the max-depth=1 command as a second edit. I also did a "sudo reboot now." Although nothing has changed with the disk usage, the Apache processes have dropped off the list and only the postmaster/Postgres ones remained. – Dave Nov 09 '19 at 00:04
  • Not sure if I understand your question correctly. Do bear in mind that / includes all directories which are not mapped to other partitions or disks. So, if you check /, it will very probably include /lib, /tmp, /var and also /usr/lib and others which are rarely mapped. There used to be a great utility called xdiskusage which graphically shows where things are. – jcoppens Nov 09 '19 at 04:30
  • @jcoppens, My question is how do I get the usage reported by "df" to go down? Utilities like "du" and "ncdu" report that "/" and its sub-directories are only using fractions of what "df" is reported being used on "/". – Dave Nov 09 '19 at 18:20
  • try running du -xh / | sort -h > /tmp/ssu to get a sorted list of the directories by space used. then examine say the tenth from the end entry (the last entry will be for "/", and probably the one before that will be "/var") – icarus Nov 10 '19 at 00:07

0 Answers0