1

I am working on a cluster that requires both public key and password, I have a complex file structure to organize my work on this cluster, and I would like to have a backup on my local machine of some directories only.

I would like rsync to get only directories that have a file named "backup". Example:

data/sub1/sub1_1/backup < back up this directory
data/sub1/sub1_2/ < don't back up

Because of this password thing, I would like to avoid any script that calls ssh multiple times, is there any advanced filter able to do that with rsync?

AdminBee
  • 22,803
Okano
  • 111
  • 1
    Open an SSH connection the first time and keep it open. See https://unix.stackexchange.com/questions/33557/using-an-already-established-ssh-channel for how to set this up (make sure you do keep one connection open, e.g. with ControlPersist). That way it doesn't matter if you invoke rsync multiple times: they'll reuse the existing SSH connection. – Gilles 'SO- stop being evil' Oct 29 '21 at 16:37

2 Answers2

0

If the number of directories to backup is not too big, you can put them into the command line using a bash shellscript. The following shellscript is a demo.

Directory structure

$ tree data
data
└── sub1
    ├── sub1_1
    │   ├── a
    │   ├── b
    │   └── backup
    ├── sub1_2
    │   └── c
    └── sub1_3
        ├── backup
        └── d

4 directories, 6 files

Shellscript rsyncer

#!/bin/bash

echo -n 'rsync -avn ' > command
find . -name 'backup' -type f | sed -e 's%/backup%%' -e 's%.*%"&"%' | tr '\n' ' ' >> command
echo ' target/' >> command


bash command

Dry run

$ ./rsyncer 
sending incremental file list
created directory target
sub1_1/
sub1_1/a
sub1_1/b
sub1_1/backup
sub1_3/
sub1_3/backup
sub1_3/d

sent 199 bytes  received 64 bytes  526.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Please notice that sub1/sub1_2 and the file c are not listed.

Backup

Remove the option n from rsync in the shellscript and run it or from the file command and run it,

sed 's/-avn/-av/' command > buper

$ bash buper
sending incremental file list
created directory target
sub1_1/
sub1_1/a
sub1_1/b
sub1_1/backup
sub1_3/
sub1_3/backup
sub1_3/d


sent 383 bytes  received 148 bytes  1,062.00 bytes/sec
total size is 0  speedup is 0.00


$ tree target
target
├── sub1_1
│   ├── a
│   ├── b
│   └── backup
└── sub1_3
    ├── backup
    └── d


2 directories, 5 files

Please notice that sub1/sub1_2 and the file c are not listed.

sudodus
  • 6,421
-1

Maybe you could use something like it.

  1. Use the find command to find files that you are interested in.
  2. Put them through dirname to print the directory name in which the file is stored.
  3. Use uniq so the directory is listed only ones
  4. Iterate over the result in rsync or put them together so rsync will be invoked only once.

Simple script:

for i in $(find . -name 'backup' -type f | xargs dirname | uniq); do
  echo "here: $i"
  # here goes rsync
done
Alex Baranowski
  • 1,181
  • Thank you for your answer, but if I do that I will need to type my password each time I call rsync – Okano Oct 29 '21 at 12:58