1

I'm trying to make a wrapper around my rsync command to reuse code. I need to pass the source folder, the target folder and then any number of excluded directories. Here is what I have with the logs; basically I am building my multiple rsync's --exclude params from my "vararg" param. There may be a better way but I found that one ok enough.

function run_backup(){
    source="$1"
    target="/backups/unraid$2"
    #Copying $source into remote folder $target
# Building my exclude params
if [ $# -ge 3 ]; then
    excludes="--exclude '$3'"
    while shift && [ -n "$3" ]; do
        echo $excludes
        excludes="${excludes} --exclude '$3'"
    done
else
    excludes = "--exclude '/tmp'" # Dummy exclude; I'll handle this case later
fi
echo $excludes

# Creating folder because rsync crash otherwise n some case
ssh -p 22 -i /home/backup/.ssh/mykey unraid@94.23.49.55 "mkdir -p $target"

# Executing the copy
set -e
sudo rsync -azvh \
    -e "ssh  -p 22-i /home/backup/.ssh/mykey "\
    --progress \
    --delete \
    $excludes \
    $source \
    unraid@69.69.69.69:$target

}

mkdir -p /tmp/dummy echo "dummy" > /tmp/dummy/save-ignored-1 echo "dummy" > /tmp/dummy/save-ignored-2 echo "dummy" > /tmp/dummy/save-test run_backup /tmp/dummy/ /dummy /tmp/dummy/save-ignored-1 /tmp/dummy/save-ignored-2 /tmp/toto /tmp/tata /tmp/tutu #TODO

--exclude '/tmp/dummy/save-ignored-1'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto' --exclude '/tmp/tata'
--exclude '/tmp/dummy/save-ignored-1' --exclude '/tmp/dummy/save-ignored-2' --exclude '/tmp/toto' --exclude '/tmp/tata' --exclude '/tmp/tutu'
++ ssh -p 22-i /home/backup/.ssh/mykey unraid@69.69.69.69 'mkdir -p /backups/unraid/dummy'
++ sudo rsync -azvh -e 'ssh  -p 22 -i /home/backup/.ssh/mykey' --progress --delete --exclude ''\''/tmp/dummy/save-ignored-1'\''' --exclude ''\''/tmp/dummy/save-ignored-2'\''' --exclude ''\''/tmp/toto'\''' --exclude ''\''/tmp/tata'\''' --exclude ''\''/tmp/tutu'\''' /tmp/dummy/ unraid@94.23.49.55:/backups/unraid/dummy

We can see the $excludes var contain exactly what I want; but bash is doing some magic on it when I pass it to rsync. And I have no idea how to avoid that. I guess it has to do with expansion, but I could not find a problem similar to mine or the revelant doc.

EDIT: code after first answer:

function run_backup(){
    excludes=()
    if [ $# -ge 3 ]; then
        echo $3
        excludes+=(--exclude $3)
        while shift && [ -n "$3" ]; do
            echo $3
            excludes+=(--exclude $3)
        done
    else
        excludes+=(--exclude /tmp)
    fi
    set -x
    echo "${excludes[@]}"
}
run_backup /tmp/dummy/ /dummy "/tmp/dummy/save-ignored" "/tmp/dummy/save non'standard ignored" /tmp/toto /tmp/tata /tmp/tutu

But it won't work for string containing quotes (at least simple quotes)

/tmp/dummy/save-ignored
/tmp/dummy/save non'standard ignored
/tmp/toto
/tmp/tata
/tmp/tutu
++ echo --exclude /tmp/dummy/save-ignored --exclude /tmp/dummy/save 'non'\''standard' ignored --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu
--exclude /tmp/dummy/save-ignored --exclude /tmp/dummy/save non'standard ignored --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu

EDIT 2: Useless; just keeping it for history A test with all quotations possibilities:

run_backup(){
    excludes=()
    excludes2=()
    excludes3=()
    if [ $# -ge 3 ]; then
        echo $3
        excludes+=(--exclude $3)
        excludes2+=(--exclude "$3")
        excludes3+=(--exclude "\"$3\"")
        while shift && [ -n "$3" ]; do
            echo $3
            excludes+=(--exclude $3)
            excludes2+=(--exclude "$3")
            excludes3+=(--exclude "\"$3\"")
        done
    else
        excludes+=(--exclude /tmp)
    fi
    set -x
    echo "${excludes[@]}"
    echo "${excludes2[@]}"
    echo "${excludes3[@]}"
     sudo rsync -azvh \
        --dry-run \
        "${excludes[@]}" \
        --dry-run \
        "${excludes2[@]}" \
        --dry-run \
        "${excludes3[@]}" \
        --dry-run \
        ${excludes[@]} \
        --dry-run \
        ${excludes2[@]} \
        --dry-run \
        ${excludes3[@]} \
        /tmp \
        unraid@69.69.69.69:/tmp
}
run_backup "/tmp/dummy/" "/dummy" "/tmp/dummy/save non'standard" "/tmp/toto"
++ echo --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto
--exclude /tmp/dummy/save non'standard --exclude /tmp/toto
++ echo --exclude '/tmp/dummy/save non'\''standard' --exclude /tmp/toto
--exclude /tmp/dummy/save non'standard --exclude /tmp/toto
++ echo --exclude '"/tmp/dummy/save non'\''standard"' --exclude '"/tmp/toto"'
--exclude "/tmp/dummy/save non'standard" --exclude "/tmp/toto"
++ sudo rsync -azvh --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude '/tmp/dummy/save non'\''standard' --exclude /tmp/toto --dry-run --exclude '"/tmp/dummy/save non'\''standard"' --exclude '"/tmp/toto"' --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude /tmp/dummy/save 'non'\''standard' --exclude /tmp/toto --dry-run --exclude '"/tmp/dummy/save' 'non'\''standard"' --exclude '"/tmp/toto"' /tmp unraid@69.69.69.69:/tmp

Screenshot of my test with better formated outputs:Screenshot of my test with better formated outputs

  • See also: http://askubuntu.com/questions/320458/how-to-exclude-multiple-directories-with-rsync & http://askubuntu.com/questions/545655/backup-your-home-directory-with-rsync-and-skip-useless-folders & http://askubuntu.com/questions/40992/what-files-and-directories-can-be-excluded-from-a-backup-of-the-home-directory/40997#40997 – oldfred Aug 31 '22 at 17:52
  • 1
    this, excludes="--exclude '$3'", puts literal quotes in the value of excludes, they'll stay literal as the shell doesn't parse the results of expansions for shell syntax (it would be a horrible safety issue, e.g. it'd never be possible to deal with strings like ain't so in a variable). Even if it did work, it'd fail for values that themselves contain single quotes. The subsequent $excludes will then wordsplit on whitespace, breaking any excluded paths that contain whitespace. Use an array instead since you're running Bash, which supports them. – ilkkachu Aug 31 '22 at 20:17
  • 1
    Also (you probably know this), there shouldn't be any spaces around the = in excludes = "--exclude '/tmp'". Also the function keyword is unnecessary and non-standard, just run_backup() { ... would do to define a function – ilkkachu Aug 31 '22 at 20:18
  • 1
    Please learn how to debug.  You should have been able to track your problem down to the excludes="--exclude '$3'" line; then you could have spared us the 30-line code block, and you would have been that much closer to asking a straightforward question — and, ideally, finding a solution yourself.  And your “EDIT 2” is terrible.  You use four arguments, totaling 64 characters, and three of them are equivalent (strings of letters and slashes), and the fourth one has two special features (space and quote).  Simplify!  You present a 500+ character output and make no attempt to analyze it.  Etc… – G-Man Says 'Reinstate Monica' Sep 04 '22 at 00:53
  • You should be using excludes+=(--exclude "$3"). Quote your shell parameter expansions, please. – muru Sep 04 '22 at 07:23
  • @G-ManSays'ReinstateMonica' I agree with some of this. My first question I did simplify from the orgiginal code, maybe I could have done more but I had no idea where the issue was comming from and didn't want to remove too much. My first edit is ok IMO. But my second edit is shit; I wrote it while being completly lost and wanted to show how no answer given where ok ina single example - until I find the answers where working but I did not understood how it actually worked. I explained that in my las comment to laktak and have now edited my question to avoid people the trouble of reading it. – Bancarel Valentin Sep 05 '22 at 09:09

1 Answers1

5

Use an array instead of a string.

For example:

opt=()
opt+=(--exclude "first")
opt+=(--exclude "another one")

set -x echo "${opt[@]}"

This will handle spaces correctly (output):

+ echo --exclude first --exclude 'another one'
laktak
  • 5,946
  • This is perfect for the use case described. But I actuallt would like to be able to handle files with spaces or quotes in their names. I have tried adding quotes (simple and/or double) at different places but couldn't find a fix. I have edited my answer with snippet you gave me and the log it produces to show you. Any idea of hoow to resolve that ? – Bancarel Valentin Aug 31 '22 at 21:54
  • @BancarelValentin it should already be handling such names. If it's not please add to your question the exact code you're now using with an explanation of what's not working – Chris Davies Aug 31 '22 at 22:30
  • Yup, I did exactly taht already if you want to have a look. – Bancarel Valentin Aug 31 '22 at 22:44
  • @BancarelValentin in your updated code you need to use quotes like in my answer, e.g. excludes+=(--exclude "$3") – laktak Sep 01 '22 at 19:38
  • @laktak with that, one of my excluse looks like this --exclude /tmp/dummy/save non'standard ignored. There is a quote messing up here – Bancarel Valentin Sep 02 '22 at 09:41
  • I should say I also tried excludes+=(--exclude "\"$3\""); with that the echo looks great --exclude "/tmp/dummy/save non'standard ignored" but not the actual rsync command: sudo rsync -azvh --delete --stats --exclude '"/tmp/dummy/save non'\''standard ignored"' – Bancarel Valentin Sep 02 '22 at 15:26
  • When I use your sample with "$3" I get this: + echo --exclude /tmp/dummy/save-ignored --exclude '/tmp/dummy/save non'\''standard ignored' --exclude /tmp/toto --exclude /tmp/tata --exclude /tmp/tutu which is correct – laktak Sep 02 '22 at 19:32
  • 1
    You are correct; I just edited my answer and was about to post more details but I just missed your last point '/tmp/dummy/save non'\''standard ignored' is indeed correct. I did not understood there was a concatenation in the middle there. – Bancarel Valentin Sep 03 '22 at 17:00