Can rsync resume after being interrupted?

Question

I used rsync to copy a large number of files, but my OS (Ubuntu) restarted unexpectedly:

sudo rsync -azvv /home/path/folder1/ /home/path/folder2

After reboot, I ran rsync again, but from the output on the terminal, I found that rsync still copied those already copied before. But I heard that rsync is able to find differences between source and destination, and therefore to just copy the differences.

Source and target are both NTFS. The source is an external HDD and target is an internal HDD.

I wonder in my case if rsync can resume what was left last time?

Answer 1

First of all, regarding the "resume" part of your question, --partial just tells the receiving end to keep partially transferred files if the sending end disappears as though they were completely transferred.

While transferring files, they are temporarily saved as hidden files in their target folders (e.g. .TheFileYouAreSending.lRWzDC), or a specifically chosen folder if you set the --partial-dir switch. When a transfer fails and --partial is not set, this hidden file will remain in the target folder under this cryptic name, but if --partial is set, the file will be renamed to the actual target file name (in this case, TheFileYouAreSending), even though the file isn't complete. The point is that you can later complete the transfer by running rsync again with either --append or --append-verify.

So, --partial doesn't itself resume a failed or cancelled transfer. To resume it, you'll have to use one of the aforementioned flags on the next run. So, if you need to make sure that the target won't ever contain files that appear to be fine but are actually incomplete, you shouldn't use --partial. Conversely, if you want to make sure you never leave behind stray failed files that are hidden in the target directory, and you know you'll be able to complete the transfer later, --partial is there to help you.

With regards to the --append switch mentioned above, this is the actual "resume" switch, and you can use it whether or not you're also using --partial. Actually, when you're using --append, no temporary files are ever created. Files are written directly to their targets. In this respect, --append gives the same result as --partial on a failed transfer, but without creating those hidden temporary files.

So, to sum up, if you're moving large files and you want the option to resume a cancelled or failed rsync operation from the exact point that rsync stopped, you need to use the --append or --append-verify switch on the next attempt.

As @Alex points out below, since version 3.0.0 rsync now has a new option, --append-verify, which behaves like --append did before that switch existed. You probably always want the behaviour of --append-verify, so check your version with rsync --version. If you're on a Mac and not using rsync from homebrew, you'll (at least up to and including El Capitan) have an older version and need to use --append rather than --append-verify. Why they didn't keep the behaviour on --append and instead named the newcomer --append-no-verify is a bit puzzling. Either way, --append on rsync before version 3 is the same as --append-verify on the newer versions.

--append-verify isn't dangerous: It will always read and compare the data on both ends and not just assume they're equal. It does this using checksums, so it's easy on the network, but it does require reading the shared amount of data on both ends of the wire before it can actually resume the transfer by appending to the target.

Second of all, you said that you "heard that rsync is able to find differences between source and destination, and therefore to just copy the differences."

That's correct, and it's called delta transfer, but it's a different thing. To enable this, you add the -c, or --checksum switch. Once this switch is used, rsync will examine files that exist on both ends of the wire. It does this in chunks, compares the checksums on both ends, and if they differ, it transfers just the differing parts of the file. But, as @Jonathan points out below, the comparison is only done when files are of the same size on both ends — different sizes will cause rsync to upload the entire file, overwriting the target with the same name.

This requires a bit of computation on both ends initially, but can be extremely efficient at reducing network load if for example you're frequently backing up very large files fixed-size files that often contain minor changes. Examples that come to mind are virtual hard drive image files used in virtual machines or iSCSI targets.

It is notable that if you use --checksum to transfer a batch of files that are completely new to the target system, rsync will still calculate their checksums on the source system before transferring them. Why I do not know :)

So, in short:

If you're often using rsync to just "move stuff from A to B" and want the option to cancel that operation and later resume it, don't use --checksum, but do use --append-verify.

If you're using rsync to back up stuff often, using --append-verify probably won't do much for you, unless you're in the habit of sending large files that continuously grow in size but are rarely modified once written. As a bonus tip, if you're backing up to storage that supports snapshotting such as btrfs or zfs, adding the --inplace switch will help you reduce snapshot sizes since changed files aren't recreated but rather the changed blocks are written directly over the old ones. This switch is also useful if you want to avoid rsync creating copies of files on the target when only minor changes have occurred.

When using --append-verify, rsync will behave just like it always does on all files that are the same size. If they differ in modification or other timestamps, it will overwrite the target with the source without scrutinizing those files further. --checksum will compare the contents (checksums) of every file pair of identical name and size.

UPDATED 2015-09-01 Changed to reflect points made by @Alex (thanks!)

UPDATED 2017-07-14 Changed to reflect points made by @Jonathan (thanks!)

Answer 2

TL;DR:

Just specify a partial directory as the rsync man pages recommends:

--partial-dir=.rsync-partial

---

Longer explanation:

There is actually a built-in feature for doing this using the --partial-dir option, which has several advantages over the --partial and --append-verify/--append alternative.

Excerpt from the rsync man pages:

--partial-dir=DIR
      A  better way to keep partial files than the --partial option is
      to specify a DIR that will be used  to  hold  the  partial  data
      (instead  of  writing  it  out to the destination file).  On the
      next transfer, rsync will use a file found in this dir  as  data
      to  speed  up  the resumption of the transfer and then delete it
      after it has served its purpose.
  Note that if --whole-file is specified (or  implied),  any  par-
  tial-dir  file  that  is  found for a file that is being updated
  will simply be removed (since rsync  is  sending  files  without
  using rsync's delta-transfer algorithm).

  Rsync will create the DIR if it is missing (just the last dir --
  not the whole path).  This makes it easy to use a relative  path
  (such  as  "--partial-dir=.rsync-partial")  to have rsync create
  the partial-directory in the destination file's  directory  when
  needed,  and  then  remove  it  again  when  the partial file is
  deleted.

  If the partial-dir value is not an absolute path, rsync will add
  an  exclude rule at the end of all your existing excludes.  This
  will prevent the sending of any partial-dir files that may exist
  on the sending side, and will also prevent the untimely deletion
  of partial-dir items on the receiving  side.   An  example:  the
  above  --partial-dir  option would add the equivalent of "-f '-p
  .rsync-partial/'" at the end of any other filter rules.

By default, rsync uses a random temporary file name which gets deleted when a transfer fails. As mentioned, using --partial you can make rsync keep the incomplete file as if it were successfully transferred, so that it is possible to later append to it using the --append-verify/--append options. However there are several reasons this is sub-optimal.

1. Your backup files may not be complete, and without checking the remote file which must still be unaltered, there's no way to know.

2. If you are attempting to use --backup and --backup-dir, you've just added a new version of this file that never even existed before to your version history.

However if we use --partial-dir, rsync will preserve the temporary partial file, and resume downloading using that partial file next time you run it, and we do not suffer from the above issues.

Answer 3

You may want to add the -P option to your command.

From the man page:

--partial By default, rsync will delete any partially transferred file if the transfer
         is interrupted. In some circumstances it is more desirable to keep partially
         transferred files. Using the --partial option tells rsync to keep the partial
         file which should make a subsequent transfer of the rest of the file much faster.
-P     The -P option is equivalent to --partial --progress.   Its  pur-
         pose  is to make it much easier to specify these two options for
         a long transfer that may be interrupted.

So instead of:

sudo rsync -azvv /home/path/folder1/ /home/path/folder2

Do:

sudo rsync -azvvP /home/path/folder1/ /home/path/folder2

Of course, if you don't want the progress updates, you can just use --partial, i.e.:

sudo rsync --partial -azvv /home/path/folder1/ /home/path/folder2

Answer 4

Arriving late to this, but I had the same question and I found a different answer.

The --partial flag ("keep partially transferred files" in rsync -h) is useful for large files, as is --append ("append data onto shorter files"), but the question is about a large number of files.

To avoid files that have already been copied use -u (or --update: "skip files that are newer on the receiver").

Answer 5

Several important rules:

1. rsync use delta-xfer algorithm to determine whether to resend the blocks differ, except with -W, --whole-file option.
2. rsync will write data into a temporary directory and move to destination when complete, except with --inplace option.
3. when delta-xfer is enabled, if you want to skip the computing of checksum of blocks of partial sent data, you can add --append option, but the sameness of partial sent data should be ensured by yourself.
4. --append implies --inplace, which itself implies --partial

In my case, I want to send incremental files without too much CPU & disk load, the command is

rsync -avPL --inplace --append --bwlimit 30m -e 'ssh -o StrictHostKeyChecking=no' <src> <dst>

Answer 6

In my case rsync was failing and quitting the executable. In which case I've used this simple bash while/do script.

#!/bin/bash
source="/tmp/source" # Change it !
destination="/tmp/destination" # Change it !
while true
do
  if rsync -avz --partial $source $destination; then # SC2181
    echo "rsync completed normally"
    exit
  else
    echo "Rsync failure. Backing off and retrying in 180s..."
    sleep 180
  fi
done

Before running the script, you have to set source and destination to values of your choice.

Answer 7

TLDR

This will resume partial transfers, and add compression for faster transfers

rsync --partial --progress --archive --compress --compress-choice=zstd --compress-level=9 --checksum-choice=xxh3 user@host:~/my_file.txt .

Flag explanation

--partial: Resume partial file transfer

--progress: Show transfer progress and ETA

--archive: Preserve file attributes

--compress: Enable compression (zlib by default)

--compress-choice=zstd: Enable zstd compression (faster and better compression than zlib)

--compress-level=9: Increase the compression level from the default of 3 (tradeoff vs maximum of 19)

--checksum-choice=xxh3: Use xxh3 hashing algorithm (very fast)

Answer 8

I think you are forcibly calling the rsync and hence all data is getting downloaded when you recall it again. use --progress option to copy only those files which are not copied and --delete option to delete any files if already copied and now it does not exist in source folder...

rsync -avz --progress --delete -e  /home/path/folder1/ /home/path/folder2

If you are using ssh to login to other system and copy the files,

rsync -avz --progress --delete -e "ssh -o UserKnownHostsFile=/dev/null -o \
StrictHostKeyChecking=no" /home/path/folder1/ /home/path/folder2

let me know if there is any mistake in my understanding of this concept...

Answer 9

For those using the GUI Grsync, the matching configuration is the following:

In the "Advanced options" tab, check (at least) the checkbox "Keep partially transferred files".

Then in the "Additionnal options" field, enter:

--append-verify

Then File -> Simulation : to check whether the transfer will work (if you encounter error you might want to check the other options selected, source & destination).
And finally File -> Execute.

With this configuration if the transfer fails, you can just close the transfer window and File -> Execute again. It will resume the transfer where it was interrupted.

Answer 10

You are copying between two local filesystems. This disables almost all the optimisations offered by rsync and implicitly enables --wholefile. Furthermore you have involved NTFS filesystems that don't necessarily keep enough of the metadata that rsync uses to check a file has been correctly copied.

Try this,

sudo rsync -rtv /home/path/folder1/ /home/path/folder2