0

I have a strange discrepancy. With "df" I see that my root disk is using almost 229G.

# df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/dm-2                   241G  229G   13G  96% /

but "du" tells me that I have only about 12GB (rounded up) in use.

# du -sh /* | grep G
3.2G    /run
2.1G    /usr
6.0G    /var

Both commands were run as root. What I could imagine is that there is "hidden" data in a directory where now an NFS share is mounted to. If this is the case. How could I find out what files are there "hidden". Or do you have other ideas why those numbers are so totally different?

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Raffael Luthiger
  • 330

2 Answers2

2

You probably have an app/process that has that file open. When you delete a file that is held open by an app the OS still sees the file size in df as the file is still open in memory.

Here is some documentation I wrote for a co worker that should get you what you need.

Truncate large open files

You have deleted files to free space but space not free afterward. Now df -lah and du -lah show different sizes

Use LSOF to get deleted but held files

lsof |grep deleted

This will show all files deleted but held open by an app.

java       2943  gateway  410w      REG              253,3  50482102     139274 /opt/span/app/node/default/var/attachments/att180368_0.part (deleted)
java       2943  gateway  411w      REG              253,3  46217973     139284 /opt/span/app/node/default/var/attachments/att182230_0.part (deleted)
java       2943  gateway  412w      REG              253,3  50483894     139280 /opt/span/app/node/default/var/attachments/att181920_0.part (deleted)

You can either restart the app to unlock files or truncate the files.

To truncate the files you will have to look at the output above to get the PID and fd (file descriptor number).

truncate the file with 

echo > /proc/PID/fd/fd_number

Example: to zero the file size of three listed above you would issue the following

echo > /proc/2943/fd/410 
echo > /proc/2943/fd/411
echo > /proc/2943/fd/412

if you have many to truncate bash to the rescue. 

for n in {410..412}; do 'echo  > /proc/2943/fd/$n'; done;

df -lah should show free space now BUT files will show under lsof |grep deleted but will be 1 in size 

java       2943  gateway  410w      REG              253,3         1     139274 /opt/span/app/node/default/var/attachments/att180368_0.part (deleted)
java       2943  gateway  411w      REG              253,3         1     139284 /opt/span/app/node/default/var/attachments/att182230_0.part (deleted)
java       2943  gateway  412w      REG              253,3         1     139280 /opt/span/app/node/default/var/attachments/att181920_0.part (deleted)

The files descriptors will be released on next reboot or restarting/reloading of the app that opened the files.

Of course you will have to adjust your commands to match the output of your locked files.

Mark Stewart
  • 746
  • Good idea. I didn't think of it. But sadly not the correct answer. I did find several files like this. But all of them belonged to elasticsearch. And after restarting elasticsearch the files didn't show up when doing a "lsof" but the size of "df" didn't change much. – Raffael Luthiger Mar 24 '20 at 12:15
  • I have seen poorly built log rotate setups do this. They needed to truncate the logs but just rotated them. The result was the logs rotated but the old file was never flushed. I don't think it tried to delete them so nothing shows up as deleted in lsof. Logrotate just renamed them and tried logging to new filename. The rename never worked and did leave ghost files that misreported in df. Didn't find it was happening until the volume alert went off as the vol was filling up from not properly rotating. It has been a long time so I don't remember the details. – Mark Stewart Mar 24 '20 at 20:44
1

In my case (and you helped me in a PM by telling me the same) it was a over-mount problem. I had some data in a directory on the partition, which du couldn't see, because a mount has hidden them.

So try to umount all mounts and look for data in the remaining directories.

Have a good time and thank you!

szhvoj
  • 26