1

I had 100% of disk use on 

/dev/xvda1

Which is the system partition, so I decided to mount

/var/lib

On another partition 

/dev/xvdf

since docker files under /var/lib were eating much space.

I have always 100% of the root partition used.

udev            2.0G   12K  2.0G   1% /dev
tmpfs           396M  336K  395M   1% /run
/dev/xvda1       99G   99G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdf       296G  765M  280G   1% /var/lib

and

lsof | grep deleted

shows nothing

And this is the output of du -sh

52M /admin
9.6M    /bin
25M /boot
12K /dev
5.9M    /etc
11M /home
0   /initrd.img
61M /lib
4.0K    /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt
4.0K    /opt
du: cannot access '/proc/3945/task/3945/fd/4': No such file or directory
du: cannot access '/proc/3945/task/3945/fdinfo/4': No such file or directory
du: cannot access '/proc/3945/fd/4': No such file or directory
du: cannot access '/proc/3945/fdinfo/4': No such file or directory
0   /proc
15M /root
336K    /run
9.4M    /sbin
4.0K    /srv
0   /status
0   /sys
8.5M    /tmp
1016M   /usr
877M    /var
0   /vmlinuz

The inode usage:

Filesystem       Inodes  IUsed    IFree IUse% Mounted on
udev             504621    397   504224    1% /dev
tmpfs            505859    315   505544    1% /run
/dev/xvda1      6553600 684402  5869198   11% /
none             505859     10   505849    1% /sys/fs/cgroup
none             505859      1   505858    1% /run/lock
none             505859      1   505858    1% /run/shm
none             505859      1   505858    1% /run/user
/dev/xvdf      19660800    202 19660598    1% /var/lib

Do you see any solution ?

4m1nh4j1
  • 1,873
  • 9
  • 29
  • 41
  • 4
    Did you move the files from /var/lib to /dev/xvdf before you mounted it in that place (or cleaned the /dev/lib directory)? If not then these files are 'hidden' behind /dev/xvdf when you mounted it to /dev/lib. This can explain the missing disk usage. – Marco May 19 '16 at 08:47

3 Answers3

2

That's because 'migrating' your /var/lib to another partition does not automatically clean your original files.

The trick would be to boot from another system (maybe a livecd), from there mount your /dev/xvda1 and delete the files in there.

Please be careful, and make sure all the files are actually available in your new partition before trying this.

quemeraisc
  • 211
  • Why do you need to boot into another system? Can't you just mount /dev/xvdf to /mnt and move the files? Like this: umount /dev/xvdf; mount /dev/xvdf /mnt; mv /var/lib/whatever* /mnt; umount /mnt; mount /dev/xvdf /var/lib – Lucas May 19 '16 at 09:27
  • Maybe yes, as long as the system doesn't need anything from /var/lib in the meantime. I'm not too sure... – quemeraisc May 19 '16 at 09:35
  • @Lucas typically, a running system will have files open in /var/lib so unmounting it isn't possible. – Chris Davies May 19 '16 at 09:39
2

As others have suggested you probably have the original files underneath /var/lib. If you don't (or can't) usefully boot the system into single user mode you can also access these with a "bind mount":

mkdir -p -m700 /mnt/dsk
mount --bind / /mnt/dsk
cd /mnt/dsk/var/lib

You are now in the "old" /var/lib that is hidden underneath the "new" /var/lib and you can either delete or move files out of here as necessary.

IMPORTANT the files underneath /mnt/dsk are the same files as underneath /. They are not copies. If you delete these files you are actually deleting files on your root filesystem.

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
0

What is the sum of / ( du -sh / )? The same as in df 99G? I thing files from /var/lib are still on /dev/xvda1 partition. They are "hidden" under mounted partition /dev/xvdf. You must boot live cd (e.g. sysrescure), then:

mkdir /test 
mount /dev/xvda1 /test

and try

du -sh /test/var/

You should see your "hidden" space. You have to remove this files.

PitMG
  • 31