Disk space is missing

Question

I've been running a program lately that creates about 100-200GB of temp files each time. When this runs I'm often now running out of disk space...but I'm really not.

When I run df -h I get:

Filesystem             Size  Used Avail Use% Mounted on
udev                    32G     0   32G   0% /dev
tmpfs                  6.3G  2.9M  6.3G   1% /run
/dev/mapper/data-root  912G  693G  174G  81% /
tmpfs                   32G  178M   32G   1% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
tmpfs                   32G     0   32G   0% /sys/fs/cgroup
/dev/loop2             163M  163M     0 100% /snap/chromium/1362
/dev/loop1             241M  241M     0 100% /snap/chromium/1373
/dev/nvme1n1p1         467M  228M  205M  53% /boot
/dev/loop4              56M   56M     0 100% /snap/core18/1932
/dev/loop0             374M  374M     0 100% /snap/anbox/186
/dev/loop5              98M   98M     0 100% /snap/core/10126
/dev/loop6              98M   98M     0 100% /snap/core/10185
/dev/loop7             147M  147M     0 100% /snap/code/47
/dev/loop3             143M  143M     0 100% /snap/code/48
/dev/loop8             166M  166M     0 100% /snap/electron-mail/26
/dev/loop9              61M   61M     0 100% /snap/gmail-desktop/16
/dev/loop10             55M   55M     0 100% /snap/gtk-common-themes/1502
/dev/loop11             93M   93M     0 100% /snap/insomnia/105
/dev/loop12             93M   93M     0 100% /snap/insomnia/109
/dev/loop13            177M  177M     0 100% /snap/postman/127
/dev/loop14             11M   11M     0 100% /snap/helm/302
/dev/loop16             92M   92M     0 100% /snap/go/6633
/dev/loop15            141M  141M     0 100% /snap/slack/30
/dev/loop17             92M   92M     0 100% /snap/go/6439
/dev/loop18            174M  174M     0 100% /snap/postman/128
/dev/loop19            113M  113M     0 100% /snap/gmail-desktop/12
/dev/loop20            163M  163M     0 100% /snap/gnome-3-28-1804/145
/dev/loop21            170M  170M     0 100% /snap/spotify/42
/dev/loop22            164M  164M     0 100% /snap/spotify/41
/dev/loop23            9.9M  9.9M     0 100% /snap/helm/292
/dev/loop24            138M  138M     0 100% /snap/slack/29
/dev/loop25            9.7M  9.7M     0 100% /snap/kubectl/1634
/dev/loop26            168M  168M     0 100% /snap/electron-mail/27
/dev/loop27             56M   56M     0 100% /snap/core18/1885
/dev/loop28            9.7M  9.7M     0 100% /snap/kubectl/1647
/dev/loop29             63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop30            162M  162M     0 100% /snap/gnome-3-28-1804/128
tmpfs                  6.3G   24K  6.3G   1% /run/user/120
tmpfs                  6.3G  124K  6.3G   1% /run/user/1001
/dev/sda1              932G  664G  268G  72% /media/work/WD Elements SE 25FE

As you can see I have ~912GB / and it's saying I've used 693G. But that is NOT the case in reality. See below, showing I'm barely using 500GB

When I run sudo lsof +L1 I get https://pastebin.com/WD5qTEYW (None of the files listed look interesting, i.e. none relate to the tmp files the program I'm running creates)

I've also tried adding touch /forcefsck and rebooting, it looks like fsck does not run on boot for whatever reason. I also tried adding fsck to the grub config, same result, it does not run:

>> sudo tune2fs -l /dev/mapper/data-root|grep check
Last checked:             Sun Nov 24 23:35:00 2019

So after Googling around and trying the above over and over I'm out of ideas. Can anyone figure out what's going on here?

Here's my OS info.

NAME="Pop!_OS"
VERSION="20.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 20.04 LTS"

HDD is a Samsung 970 EVO Plus 1 TB PCIe NVMe M.2 (2280) Internal Solid State Drive (SSD) (MZ-V7S1T0)

There's quite a lot of questions on this site about differing outputs between du and df. Have you looked through any of them? — Philip Couling, Nov 03 '20 at 09:27
@Richard There's nothing to “fix”. You have files taking up disk space. That's not something you can fix. — Gilles 'SO- stop being evil', Nov 03 '20 at 09:37
If your post contains sensitive information that should not have been posted in the first place, editing will not help as the revision history is still visible. Instead, look at this page in the help center. — AdminBee, Nov 03 '20 at 14:09
Yes, please stop vandalizing your post. If there is a problem with it, raise a custom moderator flag and let us know. This can help future visitors to the site, so we don't allow users to remove useful posts. — terdon, Nov 03 '20 at 14:25
If you need a post to be deleted, you need to explain why. We don't delete things lightly and when you post here you licence the content to the site. So, if you really need this to be deleted, please flag and explain why. — terdon, Nov 03 '20 at 17:22

score 8 · Answer 1 · edited Nov 03 '20 at 12:25

I've been running a program lately that creates about 100-200GB of temp files each time. When this runs I'm often now running out of disk space...

/dev/mapper/data-root  912G  693G  174G  81% /

You have 174GB free but you need up to 200GB, so you will run out of disk space. You need to make room.

but I'm really not.

Yes, you are. The numbers are above.

it's saying I've used 693G. But that is NOT the case in reality. See below, showing I'm barely using 500GB

No, the figure from df is accurate. It doesn't get more real than that.

That screenshot evidently only counts files that are readable from your user account, since it isn't showing a figure against /root and /lost+found which you can't read. So you have 435GB that are readable from your account, plus ~258GB that isn't. If you want to know what's using that space, you need to go digging with administrator rights. Run that GUI tool as administrator (sudo baobab) or run

sudo du -x -h / | sort -h >du.txt

Other complications are possible (in particular, you should make sure to have a 5% margin, i.e. don't fill your disk to more than 95%) but first you need to trust the correct figure (the one from df) instead of the partial figure.

@Richard Let it complete. The most likely location for files that you can't see without being root is under /var. — Gilles 'SO- stop being evil', Nov 03 '20 at 09:44
Running du as root can still miss some files, if they are still in use by a process but have been unlinked already. — Brian Bi, Nov 03 '20 at 19:38
@BrianBi Yes, that's one of the other complications I reference. It's not likely to be happening here since the used space survives a reboot. (It could be happening if something runs at boot time and creates large files which it immediately deletes, but that's unlikely.) — Gilles 'SO- stop being evil', Nov 03 '20 at 21:51

Disk space is missing

1 Answers1