5

On my system df -h reports that the root file system is full (100% Available 0) and also btrfs fi usage / tells me that all of the space is allocated. After a restart it could not even bring up the necessary services, so I'm now in recovery mode.

I tried to balance the system as shown here btrfs ERROR: error during balancing - No space left on device. For btrfs balance start -dusage=27 / I get:

Done, had to relocate 0 out of 56 chunks`

For btrfs balance start -dusage=28 / I get:

ERROR: error during balancing '/': No space left on device

So I started deleting files in /var/log/ but even after removing several 100 MB df -h reports that no single bit is available.

I suspect, that there is some service filling up the space the moment it gets free. But since I'm in recovery mode, there should not be so many services.

How can I find out, what is filling the space immediately after removing the files?

How can I ensure to free space, that can be used by btrfs to rebalance?

My system is debian 10.3 (buster), filesystem btrfs and I have docker installed, which I suspect being part of the problem but which is not running in recovery mode, so …

white_gecko
  • 211
  • 3
    Have you checked if snapshots are filling up your filesystem? That's a common btrfs issue. – eblock Apr 24 '20 at 12:51
  • Yes, there are tons of docker snapshots in /var/lib/docker/subvolumes . Can I just delete them, aren't they needed by docker, and will this solve the issue? – white_gecko Apr 24 '20 at 13:02
  • 1
    As eblock said, you need to look for snapshots and delete as many as you can. You can list subvolumes AND snapshots with this: btrfs subvol list / Don't bother using df for this; it won't help. If you have files which are heavy fragmented (virtual disk images, databases, systemd journal, etc) they can eat up your metadata chunks. I recommend adding the output of btrfs fi usage / to you question. – Emmanuel Rosa Apr 24 '20 at 13:03
  • As for docker snapshots, you should consult it's documentation about how it handles images. – Emmanuel Rosa Apr 24 '20 at 13:04
  • @white_gecko How did you end up resolving the problem? I'm facing the exact same problem. I'm in recovery mode and have seemingly no way to reclaim enough space to make my system usable again. – Mecha Oct 12 '22 at 12:16

1 Answers1

7

Historically, some out-of-space situations on btrfs required temporarily adding an additional disk to the volume, as the COW nature of btrfs may require some temporary space to save a modified version of blocks before being able to free the old version. Though, I understand that many of these situations have been fixed by the btrfs developers.

For emergency disk usage profiling on btrfs volumes, I would like to recommend btdu, a tool I've written:

https://github.com/CyberShadow/btdu

As it only needs to collect 100 samples to gain 1% resolution, it is able to identify large space hogs in seconds.

Vladimir Panteleev
  • 1,696