16

I have to move some files from one filesystem to another under Ubuntu. However, it is very important that the files never exist as partial or incomplete files at the destination, at least not under the correct file name.

So far, my only solution is to write a script that takes each file, copies it to a temporary name at the destination, then renames it (which I believe should be atomical) at the destination to the original filename and finally deletes the originating file.

However, writing and debugging a script seems like it is overkill for this task. Is there a way or tool that already does this natively?

Niels2000
  • 273
  • 8
    I sense an xy problem. Why do you need this? – henning Dec 21 '17 at 14:06
  • 3
    @henning There are lots of possible reasons, so this is a good question. For example, a program that automatically attempts to process files with a file name matching a certain regex (but can't be temporarily disabled for bureaucratic reasons). Or maybe a backup system that takes files and stores then on durable but write-only media. – wizzwizz4 Dec 21 '17 at 20:00
  • 1
    @henning I need it because I have a number of caches that read from several different storages, but one is nearing capacity and needs to be removed from service.

    Accordingly, it is necessary that each file, if it exists in any of the locations, needs to be complete and intact. Otherwise, there exists a race condition between a transfer starting and ending where a cache might grab (and retain) an incomplete file.

     – Niels2000 Dec 22 '17 at 12:43

1 Answers1

27

rsync copies to temporary filenames (e.g. see Rsync temporary file extension and rsync - does it create a temp file during transfer?) unless you use the --inplace option. It renames them only after the file has been transferred successfully. rsync also deletes any destination files that were only partially transferred (e.g. due to disk full or other error).

There is also a --remove-source-files option which deletes the source file(s) after they've been successfully transferred. See the rsync man page for more details.

Putting that all together, you could use something like:

rsync -ax --remove-source-files source/ target/

This option is particularly useful for tasks like moving files out of an "incoming" queue or similar to the directory where they will be processed.

Alternatively, if this is a once-off mirror, maybe just use rsync without the --remove-source-files option. You can delete the source files later if you want/need to.

cas
  • 78,579
  • I thought there would be a good tool for the job. Thanks! – Niels2000 Dec 21 '17 at 13:09
  • @Niels2000 I can't remember if I've ever used that rsync option before but what you wanted sounded like the kind of thing that rsync could do, and I knew it transferred files with a temp name (unless you use --inplace), so I looked in man rsync and found --remove-source-files. rsync is a good tool to check first for almost any file transfer task. – cas Dec 22 '17 at 05:58