1

Possible Duplicate:
How can I diagnose and repair missing drive space? 

root@database:/# treesize -x 2>/dev/null
1.1 GB .
757.1 MB ./usr
232.0 MB ./lib
97.2 MB ./var
29.0 MB ./boot
6.3 MB ./sbin
5.5 MB ./etc
5.4 MB ./bin
80.0 KB ./home
68.0 KB ./root
32.0 KB ./tmp
16.0 KB ./lost+found
12.0 KB ./mnt
12.0 KB ./media
4.0 KB ./srv
4.0 KB ./selinux
4.0 KB ./opt
0.0 KB ./sys
0.0 KB ./proc
0.0 KB ./dev
root@database:/# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9,2G  9,0G     0 100% /
root@database:~$ df -i /
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   57516  553284   10% /
root@database:/# du -sh /* 2>/dev/null
5,4M        /bin
29M /boot
0   /cdrom
144K        /dev
5,5M        /etc
84K /home
0   /initrd.img
0   /initrd.img.old
232M        /lib
0   /lib64
16K /lost+found
12K /media
12K /mnt
69G /opt
0   /proc
104K        /root
6,4M        /sbin
4,0K        /selinux
4,0K        /srv
0   /sys
32K /tmp
758M        /usr
98M /var
0   /vmlinuz
0   /vmlinuz.old

Contents of treesize

#/bin/sh
du -k --max-depth=1 "$@" | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

After an upgrade from Ubuntu 9 to 10 LTS this suddenly happened a few days later.

Why is / out of space when I can't find a single file taking up the space? I've already tried fsck from gparted live. I've also booted into single user mode and saved a copy of lsof output. But since the machine has rebooted multiple times I doubt it's due to open files.

I have mysqld on /opt and it had some error.logs open in /var/log, I tried shutting down everything I could, vmware tools, cron, syslog, mysql, atd, nrpe, snmpd and again checked treesize but no difference in the output.

Edit: I guess my accept-rate will have to drop even further because it looks like this is yet another question on one can answer.

My solution was to install a new VM, mount the old VMDK in the new VM and mount /opt from the old VMDK. Interestingly enough when I mounted the old / on /mnt/root I saw that the FS was still full. 

# df -h /mnt/root
Filsystem            Storlek Anvnt Tillg Anv% Monterat på
/dev/sdb1             9,2G  9,2G     0 100% /mnt/root
Community
  • 1
Stefan Midjich
  • 197
  • 1
    An utility like ncdu could give you more insight on what is using space. – Renan Oct 24 '12 at 12:46
  • you asked X amount of questions and have only selected an answer for 25% of X. You've left 75% of your questions marked as "unanswered". http://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work – h3rrmiller Oct 24 '12 at 13:29
  • can you show me the output of the mount command? and print out your partition table with fdisk – h3rrmiller Oct 24 '12 at 13:32
  • 1
    You could try bind-mounting / elsewhere - maybe some of your mount points are hiding things – Mat Oct 24 '12 at 15:20

1 Answers1

0

Is the large utilization persistent across reboots? Is it possible that there was an open file descriptor writing to a file that was no longer there? I've seen this cause filesystems to think they are 100% when they're really not. In that case, a reboot will fix it.

Also, I don't see that your treesize script will by default consider hidden (.) files.

Try a command like 

find . -x -type f -exec du -k {} \; | awk -F'  ' '{sum+=$1} {print $1"\t"$2} END{print "--------\n"sum"\tTotal K-Bytes"}'

That will provide similar output to your script, but will include a total line at the bottom that you can use to compare to what df says about the filesystem.

Tim Kennedy
  • 19,697
  • I mentioned that I have rebooted, and booted into single user mode to check lsof. And also run fsck from a live cd. I ended up just reinstalling a new VM and mounting the old VMDK /opt partition in the new VM, quick solution. – Stefan Midjich Oct 25 '12 at 07:08