1

I tried to delete the subDirectories using rm -rf .* it failed to delete the subfolders, It worked upto 5000...But it fails to delete 10000 + subdirectories at once , the I wrote a script which delete a files from a sub directories and then deletes a subDirectories.

    # loop & print a folder recusively,
print_folder_recurse() {
for i in "$1"/*;do
    if [ -d "$i" ];then
        print_folder_recurse "$i"
        #echo "dir: $i"
    elif [ -f "$i" ]; then
        #echo "file: $i"
        rm -rf $i
    fi
done
#echo "dir: $1"
if [[ "$1" != "$path" ]]; then
    #echo "dir : $1"
    rm -rf $1o
fi

}

Note: This scripts also sometimes failed...And it is not necessary always that the Directory size will be limited to 20 GB....It can grow more in future.

Any one experts in Linux who knows the better way to handle such issue ?

GAURAV JALAN
  • 11
  • note that with rm -rf .* you are trying to delete all dot files/directories including hidden ones – αғsнιη Jan 03 '21 at 17:47
  • I once managed to remove a UNIX system with rm -rf .*, as .* matches .. and so tries to remove the parent directory, and the parent of that, up to / .... Why not rm -rf *? – ridgy Jan 03 '21 at 18:11
  • The * operator is expanded by the shell, so rm -rf * is expanded to rm -rf dir1 dir2 dir3 dir4 dir5 ...... I think there is some limit on the length of a shell command line. You might instead use find . -maxdepth 1 -type d -exec rm -rf {} \;. Getting familiar with findwill help you a lot... – ridgy Jan 03 '21 at 18:19
  • @ridgy not for recent version of rm, see https://unix.stackexchange.com/q/90073/72456 – αғsнιη Jan 03 '21 at 18:33
  • 1
    @GAURAVJALAN also the use of unquoted variables in rm -rf $i and rm -rf $1o are ver unsafe; see also https://shellcheck.net – αғsнιη Jan 03 '21 at 18:37
  • 1
    Welcome @GAURAV ! Just as a tip, posting the actual output of the commands helps a lot for others to help you. You should get more and better answers by providing as much information as possible. Refer to this FAQ for some good tips on improving questions. – Zip Jan 03 '21 at 18:57

1 Answers1

0

The problem is not the size of things to delete, but their number.

Wether you try: rm -rf * or, in your script, for i in "$1"/*;do : the shell needs to expand the * and it results in thousands of filepath to be browsed and also put into memory. When the number is too great, it will either be extremely slow, or just fail (and display an error message).

You should simplify and just : rm -rf "${1:-missing_directory_name}" in your script, as the only command, directly: rm will handle itself the discovery of subfiles.

Another way would be to use find "${1:-missing_directory_name}" - type f -depth -print0 | xargs -0 -n 100 rm , calling an rm each time there are 100 filenames outputed by find, thereby limiting the number of arguments to be added to rm, whatever the exat number of files there are. And then another find afterwards, for exemple: find "${1:-missing_directory_name}" -type f -depth -print0 | xargs -n 100 rmdir to rmdir the remaining directories, starting with the "most deep ones", avoiding attempting to delete a non-empty directory first. (but the first solution does all of this with 1 rm and may prove to be the most efficient in the end)

While the script runs, you can do in another terminal window : several: df -i directory_name , and see the number of inodes decrease rapidly as files withing the directory get deleted. (the df -i will display all inodes of the filesystem containing directory_name, so it will probably not end at 0, as other files of this filesystem may be under other directories or the parent directory(s).

There are numerous others ways (on recent find, you have other ways to not use xargs, etc).

The idea stays the same: do not use shell expansion when the number of files expanded is too great.

Olivier Dulac
  • 6,285