Delete millions of files efficiently, without first generating a list of all files to be deleted

Question

I've been trying to figure out how to delete millions of files from a mounted NAS drive. The OS I'm accessing it from is RHEL 7.6.

The directory is actively being written to, with tens or hundreds of thousands of additional files per day.

When I try running a deletion technique, such as:

rsync -av --delete empty_dir/ millions_of_files_dir/

It will use lots of CPU/MEM at first, with the "sending incremental file list" message. This continues forever, with CPU/MEM usage slowly petering down to a minimal amount.

My theory is that this is happening because the file list never ends; there are endlessly more files being added.

SO: Is there a deletion technique which doesn't require a full list of files before deletion, but will "stream-delete" as it finds them?

Answer 1

Using the find command with the -delete option, files will be deleted as they are found, without building a full list:

find . -type f -delete # deletes all files in the current dir and all subdirs

Something to note is that most Linux filesystems perform poorly with that many files in a single directory. Even without building a "full" list, something will still have to list the contents of the directory, which will have terrible performance if it contains millions of files.