23

Recently I needed to delete a large number of files (over 1 million) and I read that doing:

rsync -av --delete `mktemp -d`/ ~/source && rmdir ~/source

Was one of the most optimized ways to do that, and I can vouch that it's faster than rm -rf.

I'm not an expert on the matter, but from my understanding the reason for rsync performance has something to do the way it lists files (LIFO instead of FIFO I suppose). Now, the problem is, I also need to to move a large number of files in a efficient way. After searching a bit, I found this:

rsync -av --ignore-existing --remove-source-files ~/source ~/destination

While this deletes all the moved files in ~/source, the directories remain there. Since I have a "round-robin"-like directory structure the number of files/directories is very close to 1, so I am forced to run the first command again to get rid of the directory entirely:

rsync -av --ignore-existing --remove-source-files ~/source ~/destination && \
rsync -av --delete `mktemp -d`/ ~/source && rmdir ~/source

A straight mv would finish virtually instantly, but my ~/destination directory has files that should be kept, so mv is not an option. I found the --prune-empty-dirs and --force rsync options, but neither seem to work as I would expect:

--force                 force deletion of directories even if not empty
--prune-empty-dirs      prune empty directory chains from the file-list
--remove-source-files   sender removes synchronized files (non-dirs)

Is there a way to mimic a move with rsync in one go?

Alix Axel
  • 2,929

2 Answers2

13

From zany's comment to slm's answer (Move files and delete directories with rsync?) I would recommend these 2 commands as an answer:

rsync -av --ignore-existing --remove-source-files source/ destination/ && \
find source/ -depth -type d  -empty -exec rmdir "{}" \;

The advantage is, like zany said, there is still some danger involved in using rm -rf if you don't get it right or for beginners.

I added 2 options, -depth and -empty and while I am not sure if this is really necessary, it makes the 2nd command more portable for other situations and even more safe (it still does the right thing if some directories are not empty and starts removing from the deepest point in a directory tree)

Community
  • 1
mit
  • 1,443
  • 2
    why not just -delete instead of -exec rmdir {} \;? – flying sheep Jan 10 '17 at 09:24
  • @flying_sheep see user7000's comment in sim's answer: when rsync ist interrupted you could loose files – mit Feb 08 '17 at 15:50
  • 2
    When rsync is interrupted, the find never starts or not? So the -delete in a second rsync shouldn't be a problem. But -delete also remove files, instead rmdir never delete files, only directories. Both (-delete and rmdir) check the directories if they are empty before deleting. – teekkari Apr 15 '17 at 19:59
  • Furthermore the man pages says u should use -execdir instead of -exec ... – teekkari Apr 15 '17 at 20:19
  • find -type d -delete only deletes empty directories (you can add -empty to avoid error messages, but don't have to) – xeruf May 30 '22 at 13:27
8

I found this thread over on stackoverflow titled: Deleting folders with rsync “move”?, which is asking essentially the same question. One of the answers suggested doing the rsync in 2 commands since it appears there isn't a single command that can accomplish the move/remove of the files and the source directories.

$ rsync -av --ignore-existing --remove-source-files source/ destination/ && \
  rsync -av --delete `mktemp -d`/ source/ && rmdir source/

Alternatively you can do it using this command:

$ rsync -axvvES --remove-source-files source_directory /destination/ && \
  rm -rf source_directory

Not ideal but does the job.

Community
  • 1
slm
  • 369,824
  • Yes, it was me who answered that question (and the same "solution" is also on this question). =P – Alix Axel Jun 29 '13 at 03:34
  • @AlixAxel - Ha, sorry I didn't even notice you answered that one. Oh well. Shall I delete this answer then? – slm Jun 29 '13 at 08:19
  • 3
    Using rsync to prune directories doesn't feel right and there is always danger with rm -rf. I'd recommend the 2nd step to be: find source/ -d -type d -exec rmdir {} \; – zany Dec 13 '13 at 14:10
  • 3
    I just feel compelled to add, never use -delete and --remove-source-files in a single rsync command (the above examples are fine, this is tangential). If you interrupt then re-execute what I'm describing, you'll lose transferred files. Which I've done before :( – Sridhar Sarnobat Jul 16 '15 at 19:03
  • When cleaning up with find, I prefer this: find source/ -type d -empty -delete It gives me a weird sense of reassurance that it won't just clobber everything. – greyfade Apr 05 '18 at 22:58