0

This might be related to this (though I'm not excluding folders, only their content), but I'm not sure I fully understand the provided answer.

Following Arch Wiki full system backup with rsync, I used the following backup script:

#!/bin/bash
rsync -aAXHs --info=progress2 --numeric-ids --delete --exclude={"/swapfile","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*",
"/mnt/*","/media/*","/lost+found","/home/*/.cache/mozilla/*","/home/*/.thumbnails/*"} -e ssh / login@ip.to.my.nas:/path/to/my/backup/folder/

Note that the path to my backup folder on the NAS does start and end with a /, in case that matters.

The code is run by a backup.service systemd service:

[Unit]
Description="Full system backup service on the NAS"

[Service] ExecStart=/usr/local/bin/backup.sh

Which is triggered by an enabled backup.timer:

[Unit]
Description="Backs up the full system"

[Timer] OnBootSec=3min OnUnitActiveSec=24h Unit=backup.service

[Install] WantedBy=multi-user.target

The SSH connection to the NAS (a QNAP NAS) is performed with a private/public SSH key, without a passphrase.

Now, when I sudo journalctl -u backup.service -b, I get:

Aug 17 17:21:08 Solgaleo systemd[1]: Started "Full system backup service on the NAS".
Aug 17 17:23:21 Solgaleo backup.sh[14481]: [6.3K blob data]
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/uinput, ACL_TYPE_ACCESS): Operation not s>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/dri/card0, ACL_TYPE_ACCESS): Operation no>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/controlC0, ACL_TYPE_ACCESS): Operatio>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/controlC1, ACL_TYPE_ACCESS): Operatio>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/hwC0D0, ACL_TYPE_ACCESS): Operation n>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/hwC1D0, ACL_TYPE_ACCESS): Operation n>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D10p, ACL_TYPE_ACCESS): Operatio>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D11p, ACL_TYPE_ACCESS): Operatio>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D12p, ACL_TYPE_ACCESS): Operatio>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D3p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D7p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D8p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC0D9p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC1D0c, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC1D0p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC1D1p, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/pcmC1D2c, ACL_TYPE_ACCESS): Operation>
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/seq, ACL_TYPE_ACCESS): Operation not >
Aug 17 17:23:21 Solgaleo backup.sh[14481]: rsync: set_acl: sys_acl_set_file(dev/snd/timer, ACL_TYPE_ACCESS): Operation no>
Aug 17 17:23:22 Solgaleo backup.sh[14481]: [47.9K blob data]
Aug 17 17:23:22 Solgaleo backup.sh[14481]: [48.0K blob data]
Aug 17 17:23:22 Solgaleo backup.sh[14481]: [48.0K blob data]
Aug 17 17:23:22 Solgaleo backup.sh[14481]: [48.0K blob data]
Aug 17 17:23:22 Solgaleo backup.sh[14481]: [48.0K blob data]
[...]
Aug 17 17:24:13 Solgaleo backup.sh[14481]: file has vanished: "/home/ben/.cache/mozilla/firefox/p4vxyaiw.default-release/>
[...]

That means at least the /dev and /home/*/.cache/mozilla/* are not excluded... The /swapfile is also backed up, which of course I don't want.

Did I do something wrong?

Edit:

Earlier, I posted this answer as the solution of my problem:

The answer was indeed provided here. But it's only after @MC68020 comments and reading this that I understood how rsync oddly manages exclusions.

The tricky thing with rsync is that you need to use a relative path when you are trying to exclude things, because when it tries to match the exclusions it won’t use the first part of the path for the match… it’s weird.

Say, for example, you are trying to backup /data/web/ and send it to another server, so you use a command like rsync -a /data/web/ user@server:/backups/data/web/ to make it happen… but you’d really like to skip syncing the /data/web/cache/ folder. When rsync goes to check your exclusion list for each item that it syncs, it won’t check /data/web/cache/ since your original rsync command is based in the /data/web/ folder. It’ll just check “cache/” against the list. So you’ll need to put “cache” into the list, not the full path.

So in my specific case, I just need to remove every / at the beginning of each path, so my paths are not absolute but relative... to /. The final result is:

#!/bin/bash
rsync -aAXHs --info=progress2 --numeric-ids --delete --exclude={"swapfile","dev/*","proc/*","sys/*","tmp/*","run/*","mnt/*
","media/*","lost+found","home/*/.cache/mozilla/*","home/*/.thumbnails/*","home/*/.local/share/Trash/*"} -e ssh / login@ip.to.my.nas:/path/to/my/backup/folder/

This, however, is wrong.

What a leading / means to rsync filter rules is "look at the following pattern at the beginning of the path, relative to the root of transfer".

Without the leading /, rsync matches the pattern anywhere in the path.

For instance, --exclude=tmp/*, without leading /, would exclude /tmp/*, but also /var/tmp/*, which is not what I want to do.

Rsync man page says:

The easiest way to see what name you should filter is to just look at the output when using --verbose and put a / in front of the name (use the --dry-run option if you're not yet ready to copy any files).

However, doing so does not work for me:

$ sudo rsync -av / tmp/ --dry-run --exclude={"/proc/*", "/run/*", "/dev/*"}|more
rsync: [sender] change_dir "/home/ben/tmp" failed: No such file or directory (2)
rsync: [sender] link_stat "/run/*," failed: No such file or directory (2)
sending incremental file list
created directory /dev/*}
./
bin -> usr/bin
lib -> usr/lib
lib64 -> usr/lib
sbin -> usr/bin
swapfile
boot/
boot/initramfs-linux-fallback.img
boot/initramfs-linux.img
boot/intel-ucode.img
boot/modelist.efi
boot/refind_linux.conf
boot/vmlinuz-linux
dev/
dev/autofs
dev/btrfs-control
dev/console
dev/core -> /proc/kcore
dev/cpu_dma_latency
dev/cuse
[...]

So I followed the manual (looking at the output using --verbose and adding / in front of the name, and it still does not exclude /dev/* for instance...

Ben
  • 227
  • 1
    Sorry not being capable to help you because I never run under a similar config. Anyway... for what it worth, I never rsync (-vaHXx8 --del) with the --exclude= option but preferably used --exclude-from=/BlahBlah/backup.exclude that is specifying some file enumerating the list of files to exclude. – MC68020 Aug 17 '20 at 21:02
  • Can you use wildcards in the /BlahBlah/backup.exclude file? Because I don't know what files are going to be in /dev or in firefox cache – Ben Aug 17 '20 at 21:04
  • Ha and, BTW, possibly a copy/paste pb but the closing curly bracket is missing (at least in your copy of your command) – MC68020 Aug 17 '20 at 21:08
  • 1
    It is not: [...],"/home/*/.thumbnails/*"} :) – Ben Aug 17 '20 at 21:09
  • 1
    And about wildcards in exclude file, yes indeed : https://sites.google.com/site/rsync2u/home/rsync-tutorial/the-exclude-from-option – MC68020 Aug 17 '20 at 21:11

0 Answers0