Understanding where disk space has gone?

Question

I am using Debian. df -h shows me that I'm using around 275GB:

 Filesystem                 Size  Used Avail Use% Mounted on
rootfs                     315G  274G   26G  92% /
udev                        10M     0   10M   0% /dev
tmpfs                      6.4G  200K  6.4G   1% /run
/dev/disk/by-label/DOROOT  315G  274G   26G  92% /
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                       13G  4.0K   13G   1% /run/shm

I want to work out where the 274GB has gone. Following the answers here, I can see that around 50GB is being used by the filesystem:

$ du -h /  --max-depth 3
...
51G     /

I also happen to know that I have a large Postgres database, so I can go and check how much space it's using:

$ psql 
=> SELECT pg_database.datname, pg_database_size(pg_database.datname), pg_size_pretty(pg_database_size(pg_database.datname)) FROM pg_database ORDER BY pg_database_size DESC;
   datname   | pg_database_size | pg_size_pretty
-------------+------------------+----------------
 mydatabase  |     230809349908 | 215 GB
 postgres    |          6688532 | 6532 kB
 template1   |          6570500 | 6417 kB
 template0   |          6562308 | 6409 kB

So there's about 215GB being used by Postgres, and about 50GB by the filesystem.

But how can I check where the remaining 10GB has gone?

It's not a big deal, but I'm just curious to know how one might track this down.

the rest could be DB overhead. look at the Postgres files and their sizes/usage — Skaperen, Jun 11 '15 at 10:27

score 4 · Answer 1 · answered Jun 11 '15 at 11:30

Chances are that the excess space are allocated to open but already deleted files.

du will only show those files which are accessible through the file system (ie: have a filename). When a file is deleted, its file name is removed from the directory, but the disk blocks (and the inode) get freed only when all processes close the open file handles to that file. The most trivial way to have problems with this is to delete open logs files, and expect the space to be freed. Then you either have to coerce the holding processes to close their file handles, or kill those processes (an exiting process always closes all its file handles). There are other ways a process could be using a file (running it as a binary, loading it as a shared object, mapping it into the memory space).

To see what open but deleted files are held by running processes, you might use utilities like lsof, or you might look into /proc/*. There, there's a directory for each process running, named with its PID (like: /proc/1 for PID number 1). Under the directory, a symlink called exe shows the binary running the as the process, a file called maps shows the files mapped into the memory space of the process, and a directory called fd lists all the open file handles of the process. If you see (deleted) there, that's it.

michas · Answer 2 · 2015-06-11T10:59:02.283

2

For interactively exploring the usage of your filesystem you can use tools like xdiskfree,kdiskfree or baobab.

You get basically the same information as in du but you don't have to us it for each subdirectory, but simply click on a directory to see details.

edited Jun 11 '15 at 10:59

answered Jun 11 '15 at 10:53

michas

21,510

There's also ncdu, which has the same functionality but is a non-graphical application (thus works over an SSH connection) – Wouter Verhelst Jun 11 '15 at 13:45

score 2 · Answer 3 · answered Jun 11 '15 at 11:19

As previously suggested you can use du -sh command to identify total space usage of any folder.

You will want to run this command with sudo to be sure that it captures size information for everything in the folder.

The -s option causes du to recursively check disk usage.

The -h option is to make the output human readable. When you get the hang of it, you can try running du with and without -h option and compare output. Sometimes you may find the output without -h is useful.

You can also submit wildcards to the du command to assess usages of multiple folders. So:

sudo du -sh /home/*

will give you total usages for each folder in /home

score 1 · Answer 4 · answered Jun 11 '15 at 10:28

1

    du -sh <directory>

gives details about the disk usage of the directory.

    du -h <directory>

will give disk usage of each element in the directory. You can use the above to find out which are the directories that are taking disk space.

answered Jun 11 '15 at 10:28

amisax

3,025

score 0 · Answer 5 · edited Apr 13 '17 at 12:36

Since du / reports 51GB, 51GB is what you're using. It's possible that there's some used space that you aren't seeing, such as deleted files or files hidden by a mount point (see the end of Why are there so many different ways to measure disk usage? — in fact the whole answer is probably of interest). But in your case that's unlikely.

If the PostgresQL database was inside your filesystem, then it would appear as files and it would be counted in the total. Evidently, since it's bigger than the files you see, the database is located elsewhere, outside the directory tree that you have access to.

The 315GB may be shared with other users. Or maybe the storage space is organized so that you get to use a 315GB filesystem, but you only see part of it as your directory tree, and the database is in another part that you don't see. E.g. there's a volume /clients/richard, it has two subdirectories root and database, and you're seeing only the directory tree from /clients/richard/root (i.e. your processes are chrooted there).

The remaining 10GB may be some recovery system (many VPS have this service where you can boot into a recovery system in case you need to repair your main system that isn't booting). Or it may be an emergency reserve that only root can use (search for “reserve” in Why are there so many different ways to measure disk usage?).

Understanding where disk space has gone?

5 Answers5

Linked