Faster way to delete large number of files

Question

I have a directory where lots of cached files are getting generated very quickly. Since these are very small files, it is consuming all my inodes very quickly.

Currently I am running the following command to find all the files older than 12 hours and delete them.

$ find ./cache -mtime +0.5 -exec rm {} \;

But the rate at which this command is deleting is slower than the rate at which files are being generated. Can someone tell me some alternative way to remove large number of files quickly.

Answer 1

find … -exec rm {} \; executes the rm command for each file. Even though starting a new process is pretty fast, it's still a lot slower than the mere act of deleting a file.

find … -exec rm {} + would call rm in batches, which is a lot faster: you pay the cost of running rm once per batch, and each batch performs many deletion.

Even faster is to not invoke rm at all. The find command on Linux has an action -delete to delete a matching file.

find ./cache -type f -mtime +0.5 -delete

However, if you're producing files at such a rate that find … -exec rm {} \; can't keep up, there's probably something wrong with your setup. If cache contains millions of files, you should split it into subdirectories for faster access.

Answer 2

Try using xargs:

find ./cache -mtime +0.5 -print0 | xargs -0 rm -f

Update explaination for @pradeepchhetri

If you use find with -exec, every file that find found will call rm one time. So if you found a huge of files, i.e 10000 files, you called rm 10000 times.

xargs will treat ouput of find as command argument to rm, so that, xargs will provide as many arguments as rm can handle at once, i.e rm -f file1 file2 ... So it makes less fork call, make program run faster.

Answer 3

If you just want to get rid of many files as soon as possible ls -f1 /path/to/folder/with/many/files/ | xargs rm might work okay, but better don't run it on production systems because your system might become IO issues and applications might get stuck during the delete operation.

This script works nicely for many files and should not affect the ioload of the system.

#!/bin/bash

# Path to folder with many files
FOLDER="/path/to/folder/with/many/files"

# Temporary file to store file names
FILE_FILENAMES="/tmp/filenames"

if [ -z "$FOLDER" ]; then
    echo "Prevented you from deleting everything! Correct your FOLDER variable!"
    exit 1
fi

while true; do
    FILES=$(ls -f1 $FOLDER | wc -l)
    if [ "$FILES" -gt 10000 ]; then
        printf "[%s] %s files found. going on with removing\n" "$(date)" "$FILES"
        # Create new list of files
        ls -f1 $FOLDER | head -n 5002 | tail -n 5000 > "$FILE_FILENAMES"

        if [ -s $FILE_FILENAMES ]; then
            while read FILE; do
                rm "$FOLDER/$FILE"
                sleep 0.005
            done < "$FILE_FILENAMES"
        fi
    else
        printf "[%s] script has finished, almost all files have been deleted" "$(date)"
        break
    fi
    sleep 5
done

Answer 4

Although find is the best (simplest, idiomatic) approach,

find $dir -exec rm {} +

You could move the directory aside, create a new directory (for your program), and then delete...

mv $idr old$dir && mkdir $dir && rm -rf old$dir

but maybe your problem is creating too many files. Why not change your program to append to an existing file, rather than create a new file? And then you could move this (logfile) aside, and then your program could create/append to a new file, for example,

fd = open("logfile","a+");

Answer 5

if the creation rate exceeds deletion rate you are best of by making the cache completely empty, and removing old files without any mtime evaluation

mv cache foobar
mkdir cache
# may require app restart
rm -rf foobar

Answer 6

rm -rf directory/ also works faster for billion of files in one folder. I tried that.

Answer 7

Another Linux specific solution would be to use inotify(7) facilities ; you'll detect when files are added, and then you'll immediately run something to remove the older ones.

OTOH, I guess that you might have some XY problem. Why do you have so many new files? Perhaps using sqlite, or GDBM indexed files, or some real database (e.g. PostGresQL, MariaDB, MongoDB) might be better.... Perhaps you need some version control system like git?

Answer 8

find . -name -mtime +0.5 -print -delete 

is another option to delete large number of files quickly.