2

I have an cvm, the disk of which is full in recent day. I use df -h, it says the disk is almost full:

> df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        99G   87G  7.2G  93% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   11M   16G   1% /dev/shm
tmpfs            16G   18M   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vdb1       394G  317G   58G  85% /data

But when I used du -sh to find which file should to delete to free the disk, I got the different result, which said less than 50% space had been used:

> du -sh / --exclude=/data --exclude=/proc
38G /

I also checked there were no deleted files:

> lsof | grep delete
[no output]

There were also no hidden dir in root / dir:

> ls .[^.]*
.autorelabel

And I also checked there were no swap space:

> free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       708Mi        21Gi        30Mi       8.5Gi        30Gi
Swap:            0B          0B          0B

As directed by @telcoM, I checked the file system is EXT4:

> lsblk -o +fstype
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT FSTYPE
sr0     11:0    1 18.8M  0 rom             iso9660
vda    252:0    0  100G  0 disk
└─vda1 252:1    0  100G  0 part /          ext4
vdb    252:16   0  400G  0 disk
└─vdb1 252:17   0  400G  0 part /data      ext4

As directed by @xhienne, I called the dumpe2fs:

> dumpe2fs -h /dev/vda1 | grep ^Reserv
dumpe2fs 1.42.9 (28-Dec-2013)
Reserved block count:     1100904
Reserved GDT blocks:      1017
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

What more actions should I do to free the disk?

Jason Pan
  • 121
  • Please run lsblk -o +fstype to identify the filesystem type on /dev/vda1. Some filesystems have special requirements - for example, btrfs might be in need of a btrfs balanceoperation. Also, if you ran lsof as a regular user, it will only show information about files and processes owned by you. Run it as root (e.g. sudo lsof +L1) to show deleted files system-wide. – telcoM Nov 15 '23 at 08:18
  • @telcoM Thanks. I checked the filesystem of vda1. It's ext4. I did call lsof as root rather other regular user. – Jason Pan Nov 15 '23 at 08:31
  • 1
    Have you tried to check filesystem errors with fsck? Could it be the apparent size -> https://unix.stackexchange.com/questions/173947/du-s-apparent-size-vs-du-s – pdem Nov 15 '23 at 09:46
  • @pdem I tried fsck -nf /dev/vda1 and there are many different errors: 10 Deleted inode N has zero dtime 7-Free blocks count wrong for group; 1-Free blocks count wrong; 17-Directories count wrong for group; 1-Free inodes count wrong. And the apparent size is 130T. Does this means I should fix the errors first? – Jason Pan Nov 15 '23 at 10:25
  • 2
    @Jason You are not fsck-ing a FS that is mounted, are you? This would lead to a corruption for sure. Assuming vda1 is a virtual disk for a VM: you must stop the VM in order to check the disk from outside (the VM host). If the FS still has errors, then correct them and recheck. Finally, start your VM and redo your du/df tests. BTW, do du -x instead of excluding mount points (you are probably missing /sys, /dev and others) – xhienne Nov 15 '23 at 11:16
  • Who is doing du and df? Root or a non-privileged user? Run them as root. Moreover, can you add the result of dumpe2fs -h /dev/vda1 | grep ^Reserv to your question? – xhienne Nov 15 '23 at 11:25
  • 1
    @xhienne Thanks. The exact command I called is fsck -nf /dev/vda1 > fsck.log. I got the vm from cloud, so I can only reboot it but I can't login the VM host. I can do little only inside the vm. I do these command as root inside the VM. – Jason Pan Nov 16 '23 at 01:37
  • 4
    It's entirely possible you have stuff in the /data folder that's masked by the volume mounted on it – muru Nov 16 '23 at 01:50
  • Thanks@muru. Although the problem is solve by the cloud vendor calling fsck on VM host, we still desire to hear more detail about how the /data may influent the df or du result if it's mounted from another device. Could you explain more? – Jason Pan Nov 16 '23 at 09:58
  • Hi. @xhienne I contacted the cloud vendor, and they called fsck on VM HOST to repair the filesystem error. After the simple action, my VM df came normal and when I called fsck inside my VM, there were much less error than before. I believe your comment is helpful to check a filesystem error. Could you make a summary as formal answer and I'll accept it. – Jason Pan Nov 16 '23 at 09:59
  • @Jason Glad thinks work better now. My comment is not worth being accepted as an answer. Actually, considering the cause of your issue, I'm not sure this question should stay here, even if it indeed may be helpful to others. You may want to write your own answer and accept it later, though. PS: you must NOT run fsck on a mounted ext4 filesystem (i.e. inside your VM since vda1 is mounted on /); if you have no other option, at least sync and remount / read-only before. – xhienne Nov 16 '23 at 10:08
  • @xhienne My comment may be misleading. I should say "I call fsck -nf on the mounted device rather than repairing opitions". Do you mean I should close the question because the problem is too specific but not so general? – Jason Pan Nov 16 '23 at 10:19
  • @Jason It's hard to say precisely if this question is legit. On the one hand, it may be seen as off-topic (« Questions describing a problem that can't be reproduced and seemingly went away on its own -or went away when a typo was fixed- are off-topic as they are unlikely to help future readers. »), OTOH nothing prevent such a problem from happening to others. I have suggested that this question be closed for this reason; the community will decide what to do. – xhienne Nov 16 '23 at 11:08
  • Regarding: "we still desire to hear more detail about how the /data may influent the df or du result if it's mounted from another device": It is possible that the original /data (before the mount) takes a lot of space. Since there's another device mounted on this folder, du would not be able to find those files, since it would only see the files from the mounted device. If that was the case, unmounting /data and running du could have shown if that folder was indeed responsible for the space that was previously "missing". – aviro Nov 16 '23 at 16:11
  • @JasonPan here's an example of that happening: https://unix.stackexchange.com/q/284101/70524 – muru Nov 17 '23 at 10:15

0 Answers0