2

The filesystem is ext4, the machine hasn't been rebooted in years and we don't want to do that now either.

We used to have a folder with millions of small (2-3kb of size) files. This almost broke the system so we fixed the code that was generating so many files and wrote a crontask that erased all the files within the directory (because rm wasn't working)

At first everything went smooth, you type ls and you got full list of the 4-5 remaining files.

On the next day however when I typed ls the system took forever to execute the command (it took minutes) and the system load went over 20 which scared me a lot.

It's basically like this for months now. The first time of the day when I do ls the system borderline slows to a crawl and eventually returns a list of ... 5 files and no subfolders.

I believe it's some ext4 cache, I've tried running various commands to no avail.

Is there anything else I could do to force ext4 to erase the cache.

The system is running in RAID 1 mode. Running cat /proc/mdstat shows that both drives are fully functional and synchronized. smartctl says the drive is in good health as well. hdparm returns the following

hdparm -tT /dev/sda1
/dev/sda1:
Timing cached reads:   19238 MB in  2.00 seconds = 9629.50 MB/sec
Timing buffered disk reads: 316 MB in  3.01 seconds = 104.92 MB/sec
Sk1ppeR
  • 123

1 Answers1

4

This is a known problem with file systems in the Ext family; see Why directory with large amounts of entries does not shrink in size after entries are removed? for details.

The only way to fix this is to re-create the directory. First, rename the existing directory (this will avoid problems with processes attempting to open files there):

mv brokendir repairdir/

Then, create a new directory (not using the old name, yet):

mkdir newdir

Move all the contents of the broken directory to the new directory:

mv repairdir/* newdir/
mv repairdir/.[!.]* newdir/
mv repairdir/..?* newdir/

(as three separate commands so that you know exactly what’s going on if one of them fails, e.g. if there are no hidden files to move).

You may want to ensure the new directory’s metadata is identical to the original’s, in particular its ownership and permissions; if you’re using GNU coreutils, this can be done (once repairdir is empty) with

cp -aT repairdir newdir

Finally, move everything back into place, and delete the old directory:

mv newdir brokendir/
rmdir repairdir
Stephen Kitt
  • 434,908