Bash : Give builded array to function as a list of argument?

Question

I have this issue with borgbackup, but because the reaction is the same, I will use rsync in my example.

I want to build an array of arguments by adding a prefix to each, and then give that array to rsync. But rsync acts like it doesn't exist.

With this script:

#!/usr/bin/env bash
#

declare -a exclude_String
for excludestr in $(cat ./list); do
    exclude_String+=(--exclude=$excludestr)
done
rsync "${exclude_String[@]}" . $Destination

and ./list:

'/home/*'
'*.vim*'

A ps of the process running does shows the argument correctly:

/usr/bin/rsync --exclude='*.vim*' --exclude='/home/*' . DESTINATION

But rsync still acts like they aren't there. I found this question on the subject, so I tried:

#!/usr/bin/env bash
#

declare -a exclude_String
exclude_String+=(--exclude='/home/*')
exclude_String+=(--exclude='*.vim*')

rsync "${exclude_String[@]}" . $Destination

And this actually worked.

I don't understand much yet about shell expansion and stuff, I've tried to quote and double quote all over the place just to see if I could find something, but no luck.

Any idea on how to achieve that?

Answer 1

If the file actually contains '/home/*', with the quotes, the quotes will be there in the value of excludestr in the loop, and then in the argument you give to rsync. They're not part of a shell command in that case, just part of the expanded value of the command substitution.

On the other hand, if you write exclude_String+=(--exclude='/home/*'), now the quotes are part of a shell command, and are removed during the processing of that command. (With the effect of making * an ordinary character.)

rsync doesn't actually expect or process quotes as part of the exclusion pattern, it will just treat them as any other character, and would exclude files with names containing quotes. So, if you have the exclusion patterns as data in some file, you shouldn't have the quotes there.

Also, instead of for x in $(cat file), use while IFS= read -r line; do ...; done < file, or in this case, mapfile -t exclude_String < file. That command substitution with cat invokes word splitting, so you can't have exclusion patterns containing whitespace. It also invokes globbing of the patterns immediately, right there, on the list of words for the for loop. This isn't what you want.

---

In general, what you need is:

excludes=()
while IFS= read -r line; do 
    excludes+=(--exclude="$line")
done < ./list 
rsync "$flags" "${excludes[@]}" . "$destination"

With ./list containing just the patterns to exclude:

/home/*
*.vim*

(The quotes around "$line" in +=(--exclude="$line") prevent the shell from globbing files called --exclude=something in the current directory.)

Though in the case of rsync, you might as well use --exclude-from:

rsync "$flags" --exclude-from=./list . "$destination"

If you're going to use something more complicated than a file to feed the loop, you can use process substitution:

while IFS= read -r line; do
    excludes+=(--exclude="$line")
done < <(some command that produces the exclude patterns)

See also: Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

Answer 2

As ilkkachu explained, your issue is that the single quotes should not be part of the patterns and that if you write the patterns in a shell context, the shell will remove them.

However, it looks like you can bypass having to read you patterns explicitly from your file:

rsync --exclude-from=./list source/ target

With --exclude-from=FILENAME, rsync will read the exclusion patterns from the named FILENAME. Note that when doing this, the patterns should not have those single quotes in them.