10

The usual way to get directory size is:

du -hs /path/to/directory

However, it's not working and I suspect it's a btrfs issue. It's reporting that the size of a subdirectory with submodules is more than the total size of the disk itself.

However I'm able to check using gnome disks utility that the disk is more than 60% empty.

On further inspection by varying n in 

du -h --max-depth=n /path/to/folder

I'm able to narrow large subdirectory sizes to the btrfs subvolumes which is why I think this issue is related to subvolumes

Peeyush Kushwaha
  • 516

2 Answers2

10

Yes, it's probably a BTRFS related issue.

As a general rule, classic du is not reliable with BTRFS, and quite often can give seemingly nonsensical results like this. This ultimately arises from the fact that stat() (the system call that du uses to see how much space each file is using on-disk) doesn't know or care about reflinks, snapshots, transparent compression, or pretty much anything else that BTRFS provides beyond the standard POSIX semantics.

The impact of this is that du will count any blocks shared between files via reflinks (which includes any blocks that have been deduplicated, blocks shared because of files being copied using the clone ioctl, and any that are part of snapshots) once fore each file that holds a reference to that block, and therefore du can show apparent disk usage far greater than actual space usage.

I would suggest using btrfs filesystem du instead of du, as it will properly count all that shared space usage.

Note also that GNOME Disk Utility and df are also not entirely reliable when dealing with BTRFS, but for different reasons (they properly account for reflinks, but don't understand the two-stage allocator BTRFS uses, so they can actually report the disk as mostly empty even if you can't write anything to it). As a result of this, they will quite often show different usage values from those you would get by running du on the root of the filesystem. (and it gets even worse if you use du -x, because that won't cross subvolume boundaries since they look like mount points).

Austin Hemmelgarn
  • 11,867
  • I'm not seeing any du in suggestions on btrfs filesystem <command>. I can't find it in my man page either, but I see that it's on https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-filesystem . Do I need to have some minimum kernel version / install something external for this command? – Peeyush Kushwaha Apr 09 '18 at 19:34
  • @PeeyushKushwaha You've just got an older version of btrfs-progs that pre-dates the addition of the filesystem du sub-command. Provided your distribution has newer packages, you should be able to safely upgrade just that package without having to worry about the kernel version (though making sure you're running the latest kernel is still a good idea, bugs are still being found and fixed on a somewhat regular basis in BTRFS). – Austin Hemmelgarn Apr 10 '18 at 18:16
2

As mentioned by Austin, tools such as du can't account for extents shared by files and subvolumes. For this purpose, a btrfs-aware tool is needed.

I have written a tool for exactly this purpose:

https://github.com/CyberShadow/btdu

btdu will be able to show the total amount of unique data used by a directory containing subvolumes or other constructs which share files. To use, run sudo btdu /path/to/your/volume/root, wait until it accumulates sufficient data, then descend to the desired directory.

Vladimir Panteleev
  • 1,696