CentOS 6 ran out of space, need to reclaim it

Question

Having an issue here with CentOS 6 that shows all of the space used up, but I can't tell where the space went.

Seemingly I am using up 100%, according to

df -h

Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_sugar-lv_root   50G   49G     0 100% /

So that's 49G out of 50G, fair enough, but when I try to see what exactly is being used (x to remove mounts):

du -xsh /*

1.8G    /usr
2.0G    /var

That's all, these are the 2 biggest directories, all others combined are under 1G.

Here is info on Disk /dev/mapper/vg_sugar-lv_root, it does show close to 50G (this is a virtual machine):

fdisk -l

Disk /dev/mapper/vg_sugar-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Node count is at 7%:

df -i

Filesystem                   Inodes  IUsed     IFree   IUse% Mounted on
/dev/mapper/vg_sugar-lv_root 3276800 200552    3076248    7% /

The system was rebooted several times, so any log files should've been deleted. I'd appreciate some tips in resolving this.

Answer 1

Ok, so I'll post the solution. In this case, it was a network location /mnt/backup mounted over a local directory /mnt/backup. once unmounted with

umount /mnt/backup

it showed the local directory that was taking up 45G, filled with backups:

cd /mnt/backup/ du -h 39G ./servers-unix-hq/sugar.gnsa.local 39G ./servers-unix-hq 4.5G ./db-mysql-hq/sugar.gnsa.local 4.5G ./db-mysql-hq 44G I've removed some old backups and MySQL is able to start now.

Answer 2

This is often caused by someone deleting (rather than truncating) a file that is growing. The disk space will only be released after the program writing to the file closes it.

Use lsof +L1 to get a listing of all deleted-but-still-open files, and the processes holding them open. The output will look like this:

# lsof +L1
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
php-fpm7.  1364     root    3u   REG   0,45        0     0   41658 /tmp/.ZendSem.plytyh (deleted)

The COMMAND and PID fields will tell you which process is holding the file open, and NAME indicates what the name of the file used to be. The TYPE field has a value of REG, indicating it's a regular file. The SIZE/OFF value may give you a clue of the size of the deleted file, but it is not always reliable (only when the program keeps appending to the end of the file and not updating anything in the middle, which is fortunately the default with most log files).

Once you've identified the cause of the problem, there are a few possible ways to fix it:

1. Invoke the log rotation feature of the process in question, if possible

If the problem file is a logfile and the process holding it open has a feature that allows log rotation (i.e. the process can be told to close and reopen all its log files), you can just trigger the log rotation feature. Many daemon processes accept a kill -HUP or some other signal for this purpose. The data within the deleted file will be lost unless you recover it first (see below).

2. Stop and restart the process holding the file open

Stopping and restarting the process holding the file open will also fix it and cause the contents of the deleted file be permanently lost (unless recovered first, see below).

3. (workaround) Access the deleted-but-still-open file to truncate it to zero size

Using the PID and FD numbers in the lsof +L1 output, if you have root access you can still access the "deleted-but-still-open" file through the /proc filesystem. The file in my example above would be accessible as /proc/1364/fd/3.

(Any letters after the FD number are describing how the process is accessing the file, and can be ignored.)

If you need to recover the contents of the deleted file, you can just cat it and pipe the output to another file (on another filesystem with enough free space!).

If you just need to release the disk space, you can truncate the file to zero size with command > /proc/1364/fd/3 (i.e. a command line with no command, just an output redirection to the file to be truncated).

Note: this just clears away the existing data. It will not stop more data accumulating into the deleted file, but may allow you to e.g. keep a critical service running until maintenance downtime can be scheduled.

---

Deleting a log file while it's still being written to is a common mistake by junior Unix sysadmins. Just about everyone will make this mistake at least once in their Unix/Linux career. Good ones will learn from the mistake, both to troubleshoot it as it happens, and how to avoid it in the future.