How to troubleshoot disk is full

Question

My disk drive /dev/sdc is full (see below). Is there anyway to troubleshoot why it consumed all the disk space.

I have tried running du -d -h 1 but I don't think it's the proper command.

$ df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         14G   13G     0 100% /            <--- here
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  174M  1.7G  10% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1       488M  132M  321M  30% /boot
/dev/sdd         99G   25G   73G  26% /data
tmpfs           378M     0  378M   0% /run/user/0

Other log outputs:

$ du -h -d 1

52K     ./tmp
131M    ./boot
24M     ./opt
4.0K    ./mnt
34M     ./etc
1.6G    ./root
0       ./sys
44K     ./sysadmin
du: cannot access './proc/58885/task/58885/fd/4': No such file or directory
du: cannot access './proc/58885/task/58885/fdinfo/4': No such file or directory
du: cannot access './proc/58885/fd/4': No such file or directory
du: cannot access './proc/58885/fdinfo/4': No such file or directory
0       ./proc
16K     ./lost+found
0       ./dev
25G     ./data
4.0K    ./media
32K     ./home
182M    ./run
2.6G    ./usr
921M    ./var
4.0K    ./srv
31G     .

$ du -ahx --threshold=1G

1.2G    ./root/ffmpeg_sources
1.6G    ./root
2.6G    ./usr
5.2G

Answer 1

What I would do first is trying to identify what filled the drive. According to your mounts I would start checking /home, /usr or /var (/var keeps logs, maybe a process is generating too much error messages in your logs) disk usage, once you find a directory that took your space go down and find subdirs.

Once you nailed it you can see if it was your problem (you downloaded too much vids and filled the whole drive) or if it is a process you have running creating stuff like logs, data files, etc...

If the problem wasn't you and was a process you will have to identify it with the information you have gathered from the previous phase and now you may be able to solve your problem.

EDIT: Reviewing my answer I can see I missed an importan point, if you have files created on a directory wich is a mount point and then you mount something on to that directory you won't be able to see those files unless you umount your device from that directory. You might have files in there that you are not aware off, so you might need to unmount devices and see if the directory have files there.

Answer 2

There are a number of approaches you need to take in order to identify unexpected use of disk.

1. Hidden disk use

   Several people have suggested du -xhs /* or some variant thereof. It's a good starting point but it doesn't look under mount points (where you might have inadvertently written data and then mounted a filesystem on top of it). The command also reports on filesystems other than the root because of the inclusivity asked by the wildcard.

   You can get around these issues with a bind mount.

   mkdir -p /mnt/root
   mount --bind / /mnt/root
   shopt -s dotglob
   du -hs /mnt/root/*
   

   This mounts the root filesystem under /mnt/root and reports a summary for all its top-level directories (and files). Unlike the primary mount at / there are no subsidiary mounted filesystems to consider (and avoid).

2. Deleted files

   Files that have deleted but are still open retain their disk allocation until they are closed. This has been covered in another answer so I'm only going to mention it here for completeness.

   lsof | grep deleted
   

Answer 3

One reason that drives can fill without showing evidence of it that du can see relates to private files. A process can open a file to write, begin writing, and then delete the file from its directory. This leaves an open file that the process may still write to, that still consumes disk space, but that is undiscoverable by browsing the filesystem. This sounds dumb, but it can be very useful for allocating temporary storage for sensitive information that you don't want exposed to other processes.

Other programs can also delete such files, and the program that has the file open to write is none the wiser. Sometimes users try to free space by deleting log files (or other files) that the owner process (e.g. syslogd) still have open. The result is that the files continue occupying space and even growing, but they becomes harder to find.

To find such files, you have to look at the kernel's open files list. lsof is the tool of choice for this. It shows all open files, and the offset into the file that the kernel is currently reading from or writing to.

Here's a command that might help:

lsof -o /dev/sdc | grep deleted

Let's understand what this is doing:

• lsof lists open files
• -o means to always show file offsets
• /dev/sdc limits results to files that are on the /dev/sdc block device

When a file is deleted, lsof will show (deleted) at the end of the line, after the location where the file used to exist. grep deleted finds those.

Some of these files can be normal. The 7th column of output shows the offset, though; it's prefixed with 0t. If you see an offset with a large number that's also deleted, it's consuming a bunch of disk space that you can't see.

To release that space, you must kill the process. (Depending on the process, you might want to restart it.) You should see disk space rise soon after.