Usage of --remove-source-files option of rsync

Question

From the manpage of rsync

--remove-source-files

This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the transfer and have been successfully duplicated on the receiving side.

• Does it mean files on the sending side that are either part of the transfer or duplicated on the receiving side?

• Can I also remove directories on the sending side?

Note that you should only use this option on source files that are quiescent.

• What does "source files that are quiescent" mean?

If you are using this to move files that show up in a particular directory over to another host, make sure that the finished files get renamed into the source directory, not directly written into it, so that rsync can't possibly transfer a file that is not yet fully written.

• What does this mean?

If you can't first write the files into a different directory, you should use a naming idiom that lets rsync avoid transferring files that are not yet finished (e.g. name the file "foo.new" when it is written, rename it to "foo" when it is done, and then use the option --exclude='*.new' for the rsync transfer).

• What does this mean?

Starting with 3.1.0, rsync will skip the sender-side removal (and output an error) if the file's size or modify time has not stayed unchanged.

• What does this mean?

Thanks.

Answer 1

Q: Does it mean files on the sending side that are either part of the transfer or duplicated on the receiving side?

• A: Both

Q: Can I also remove directories on the sending side?

• A: Yes
  --remove-source-files then issue the command
  find <source_directory> -type d -empty -delete OR
  find <source_directory> -type l -type d -empty -delete (to include symlinks in the deletion)

(Was: --remove-source-files then issue the command rm -rf <source_directory>)

---

WARNING:

As mentioned in OrangeDog's comment, the rm -rf suggestion is unsafe. Specifically, any files that were for any reason not transferred (file changed between building the transfer list and starting to actually transfer that file, receiving side ran out of disk space, network connection dropped, etc.) will be left untouched in the source directory by rsync — but after your rm -rf invokation they're just gone. The find command above will recursively delete the empty source tree if all the source files have been successfully transferred and removed, but will leave alone any remaining files (and their containing directories, of course).

---

Q: What does "source files that are quiescent" mean?

• A: It means files that have been written to and closed

Q: If you are using this to move files that show up in a particular directory over to another host, make sure that the finished files get renamed into the source directory, not directly written into it, so that rsync can't possibly transfer a file that is not yet fully written. What does this mean?

• A: It means exactly what I said above

Q: If you can't first write the files into a different directory, you should use a naming idiom that lets rsync avoid transferring files that are not yet finished (e.g. name the file "foo.new" when it is written, rename it to "foo" when it is done, and then use the option --exclude='*.new' for the rsync transfer). What does this mean?

• A: It means that RSYNC makes a list of files to be transferred first. Then it writes them into a different directory (Destination Directory), thus if you transfer a file that hasn't finished, it is best to rename it after it is done using the --exclude option

Q: Starting with 3.1.0, rsync will skip the sender-side removal (and output an error) if the file's size or modify time has not stayed unchanged. What does this mean?

• A: If RSYNC detects that when its about to write the file to the destination directory that the file size has changed between the time it scanned it, to the time it actually writes it to the destination directory, then RSYNC will skip the file.

Answer 2

Contrary to another answer on this question, it can be safe to mix --remove-source-files and rm -rf under one certain condition.

However, let's back up a little to respond to your other specific questions.

The --remove-source-files flag will arrange for files to be removed after they have been correctly and completely copied to the destination. They might have been copied during this session or during some possible earlier session; it doesn't matter.

The flag does not remove directories on the sending side, only files. The usual approach is to naïvely call rm -rf. This can be disastrous in the event of a failed transfer. The solution here is to ensure that it is only called once rsync has completely and correctly transferred all the files

rsync -a src/ dst && rm -rf src

The caveat is that there is an implicit race condition here. If a file in the src tree is created or modified after the rsync has processed it, it will be silently and completely deleted by the rm despite no longer being correctly synchronised to the remote system. If you control the files in your src tree and can be assured this will not occur you are safe to use this rsync && rm construct.

Newer versions of rsync can detect files that have changed during their copy. (Not during the entire copy of src, but just during the copy of src/.../file.) They do this by comparing the size and modification time of the file at the start and end of the file copy. It is an error if the file has changed.