See only changes when running rsync

Question

I am using rsync to backup files from mac to external hard drive. For example:

rsync -av --delete ~/Pictures/ "/Volumes/My Passport/Mac Backups/Pictures"

Before I do it, I run the same command in dry-run, by adding -n option:

rsync -avn --delete ~/Pictures/ "/Volumes/My Passport/Mac Backups/Pictures"

As I understand, I should see only the difference what will be copied/changed, but for some reason all folders and files are getting printed out. Though if I test the same command locally with 2 folders, I can see only the difference what has changed. Why it is so and how can I fix it?

Updated:

After adding -i option (thanks to Learner answer) I was able to identify why all the files are getting listed. It seems that permissions are not being copied. All folders (and files) have this:

.d...p... my folder/

I added -p (and -o, -g) option that should copy the permissions, but still no luck. Any ideas?

Answer 1

There is -i option for rsync, that stands for --itemize-changes. Manual to rsync is saying that it will "output a change-summary for all updates".

It will show you which files and directories has been changed and therefore transferred to the target directory from source directory. It will show you also why rsync assumed that file have been changed, for example if it's because source file size differs from the target file, or it's timestamp differs, or maybe just permissions.

Here is an example. I created directory1 and directory2. Both contain file called textfile and both files have just "A" letter inside. But file in directory1 have different timestamp. So, executing rsync -ahzi directory1/ directory2/ gives me:

.d..t...... ./
>f..t...... textfile

> means that the file was transferred to local target because it was different (if you will rsync files to 192.168.1.100:/some/directory for example, you would see <, because then it would be a remote target). f stands for "file", and t means that the timestamp of that file was different.

Now what if the content of the file changed as well? I edited textfile in directory1 so it contains letters "AB". Then I executed rsync -ahzi directory1/ directory2/:

.d..t...... ./
>f.st...... textfile

There is > showing us that files was transferred to the target because it was different. There is s telling us that size of file was different, and t saying us that timestamp was different as well.

Other "flags" that you could see in that output include:

p - Permission are different
o - Owner is different
g - Group is different
a - The ACL information changed

My source where I got explanation for all those flags: http://andreafrancia.blogspot.com/2010/03/as-you-may-know-rsyncs-delete-options.html

Answer 2

As Thomas pointed out the problem is to do with the Format of the hard drives (proposed solution --chmod didn't work for me). My external hdd is ExFAT and Mac's hdd is Mac OS Extended. So I did some googling and found the solution here >>

a: archive, replaces the rlptgoD switches (recurse dirs, preserve symlinks, preserve permissions, preserves modification times, preserve groups, preserve owner and preserve Device files).

The problem is that the Linux exFAT does not cope well with the switches that relate to permissions (the pgo), so the solution is to run rsync with the following switches, removing the p, g and o:

So the answer to my question is:

rsync -rltDvn --delete ~/Pictures/ "/Volumes/My Passport/Mac Backups/Pictures"

Now I can see only changed files. Thanks everyone for the help.

Answer 3

The likely problem is that the timestamp granularity differs on the external drive, and rsync isn't taking that into account. You may work around this using the --modify-window option:

--modify-window
When comparing two timestamps rsync treats the timestamps as being equal if they are within the value of modify_window. This is normally zero, but you might find it useful to set this to a larger value in some situations. In particular, when transferring to/from FAT filesystems which cannot represent times with a 1 second resolution this option is useful.

Reflecting the modified question: look in the rsync manual page for details on the bits for --itemize-changes. Only permissions are left to address. Most of the easy-to-use external drives are vfat: Microsoft-formatted, your Unix file-permissions won't match (because there is no analogy for the group and other permissions, and also because most applications don't bother to try to map the execute-permission to match Unix). In that case, you cannot copy the permissions. You can work around that using the --chmod option (by supplying a mask which tells rsync what permissions it can expect on the destination):

--chmod
This option tells rsync to apply one or more comma-separated "chmod" strings to the permission of the files in the transfer. The resulting value is treated as though it was the permissions that the sending side supplied for the file, which means that this option can seem to have no effect on existing files if --perms is not enabled.