2

For rsyncs that involve a large number of potentially transferred files, I mean to have a "clean" log, with only what was transferred.

I worked with stdout, since:

  1. I did not know how to configure the log output (if that is even possible).
  2. I also want this to work with --dry-run (which forced me to work with something like this, since).
  3. I mean to write scripts useful for transferring Linux-Linux, and Linux-msys2 (and viceversa) at least. Here, options for permissions (-p), owner (-o) and group (-g) may complicate things.

I combined command

rsync --recursive --perms --group --owner --times --one-file-system --sparse --stats --progress --update --out-format='%t %p %i %n %M %l' --delete --dry-run --include="*" --exclude="*" --log-file=rsync.log ./ ../trg

with

... | grep -E -v ' \.[fdLDS][\.c][\.s][\.t][\.p][\.o][\.g][\.u][\.a][\.x]'

following the description here. The first dot matches any item that is not being updated. The second character matches any item type. The remaining characters match any condition on the attributes (change/no change) Flag -v reverts matching. This presumably matches any item that is being updated, and only that. Items not updated, but with changed timestamp, would not be matched.

I finally combine this with tee, as in

rsync --recursive --perms --group --owner --times --one-file-system --sparse --stats --progress --update --out-format='%t %p %i %n %M %l' --delete --dry-run --include="*" --exclude="*" --log-file=rsync.log ./ ../trg | grep -E -v ' \.[fdLDS][\.c][\.s][\.t][\.p][\.o][\.g][\.u][\.a][\.x]' | tee rsync.2.log

which produces the rsync log file rsync.log, and my "custom" log file rsync.2.log.

Even if it seems to work, the 3 "weak" points of my approach are the following:

  1. I am not sure there are cases I am missing, where this approach might fail (by "excessive" reporting or filtering files).
  2. I would like to handle this natively via rsync itself. Even better, getting the output in the rsync log file (not only its output), rather than using the brute force grep approach. Note that using --itemize-changes I get in stdout some files with e.g. status .f...p....., which should be filtered (as they are not updated, first character is a dot).
  3. Using grep I cannot see the partial progress. I am not sure if for very large output this can be a major problem.

Related:

  1. how to make rsync to log deleted files
  2. How to make rsync display only files being copied, not all the folders which are scanned?
  3. Logging only transferred files with rsync
  4. https://superuser.com/questions/1002074/linux-command-line-to-create-a-log-file-for-rsync
  5. https://serverfault.com/questions/401210/rsync-report-only-uploaded-files

Note: The subtleties in this question (see also answer) make it slightly different from other similar questions posted/linked. Whether this warrants a separate question is a blurred line, and subjective. I personally find it useful to leave it as it is now.

AdminBee
  • 22,803

1 Answers1

0

I had 3 requirements:

  1. Configure log output. For this, --itemize-changes, as in links I had posted, and others added by @roaima, works well.
  2. Work with --dry-run. For this, --itemize-changes is still useful to produce the intended output, albeit only in stdout, and not in the log file. There seems to be no alternative.
  3. Work with various combinations of OS (src/trg). Linux-Linux, and Linux-msys2 (and viceversa), at least. In the case of Msys2, being NTFS, permissions can't match those of an ext4. So already rsynced files will keep showing, producing very long lists. In this case, --perms should be removed. Note that this would not replicate read-only <-> read-write permission changes either; I don't have a neat solution for all cases so far. Other than this, --itemize-changes is still useful in this case.