How would you gracefully handle this snippet to allow for spaces in directories?

Question

I have a series of commands, e.g.:

ssh -i key 10.10.10.10 mkdir -p "${DEST_PATH}/subdir1" "${DEST_PATH}/subdir2"
rsync "${SOURCE_PATH}" "$DEST_HOST:${DEST_PATH}/subdir1"
...

These are fed variables as

DEST_PATH=$( myfun "foo" )
SOURCE_PATH=$( myfun "bar" )

which in turn are fed string as

function myfun {
  local VALUE=$( grep "$1" yadayada.txt | tail -1 | sed 's/someregex//' )
  echo "$VALUE"
}

How would you deal with $VALUE having some spaces.. like if $VALUE was Hello World ? I thought always using "" was going to solve all issues but I was wrong. Several other questions suggest either escaping or using single quotes, but I'm not sure of the best way to go about it.

Thank you

Stéphane Chazelas · Accepted Answer · 2021-01-20T09:13:01.973

If you know the remote system has a xargs command that supports a -0 option, you can do:

printf '%s\0' "$DEST_PATH/subdir1" "$DEST_PATH/subdir2" |
  ssh -i key 10.10.10.10 'xargs -0 mkdir -p --'

That xargs -0 mkdir -p -- piece of shell code would be interpreted the same by all shells (remember sshd on the remote machine runs the login shell of the remote user to interpret the code given to ssh which could be anything).

The list of arguments for mkdir is passed NUL delimited via the remote shell's stdin, inherited by xargs.

Another advantage with that approach is that you can pass any number of arguments. If needed xargs will break that list and run several invocations of mkdir -p -- dirn dirn+1... to bypass the limit of the size of the argument+environment list, even possibly starting the directory creation before the full list has been transferred for very large lists.

See How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user? for other approaches.

rsync itself by default has the same problem as it passes the name of the remote file/dir to sync from/to as part of that remote shell command line, and the better solution again is to take that remote shell out of the loop by using the -s/--protect-args which instead of passing the file/dir names in the shell code, passes it on the rsync server's stdin.

So:

rsync -s -- "$SOURCE_PATH" "$DEST_HOST:$DEST_PATH/subdir1/"

Quoting the rsync man page:

If you need to transfer a filename that contains whitespace, you can either specify the --protect-args (-s) option, or you’ll need to escape the whitespace in a way that the remote shell will understand. For instance:
     rsync -av host:'file\ name\ with\ spaces' /dest

$ rsync --rsync-path='set -x; rsync' /etc/issue localhost:'1 2'
+zsh:1> rsync --server -e.LsfxC . 1 2
$ rsync --rsync-path='set -x; rsync' /etc/issue localhost:'1;uname>&2'
+zsh:1> rsync --server -e.LsfxC . 1
+zsh:1> uname
Linux

See how the file name is interpreted as shell code.

With -s:

~$ rsync -s --rsync-path='set -x; rsync' /etc/issue localhost:'1;uname>&2'
+zsh:1> rsync --server -se.LsfxC

no trace of the file names on the remote shell command line.

~$ cat '1;uname>&2'
Ubuntu 20.04.1 LTS \n \l

Thank you @Stéphane; I'm reading your comment talking about rsync -- how does this solution fit with rsync? If I understood correctly this will only help with executing remote commands but not with rsync, right? For rsync you advise using my code as is, but with the --protect-args flag? Thanks! — JoeSlav, Jan 19 '21 at 22:02

score 4 · Answer 2 · edited Jan 19 '21 at 21:12

Using double quotes solves the most common issues, but it doesn't solve everything. In your case, double quotes protect against unwanted expansion in the local shell, but then your ssh command injects the file name directly into a shell snippet without protection, and rsync does the same under the hood for the remote file names passed on its command line¹. That is, if DEST_PATH is /path/with three spaces, you're running the command

mkdir -p /path/with three  spaces/subdir1 /path/with three  spaces/subdir2

on the remote machine. The double quotes around $DEST_PATH ensure that the shell that's running your script doesn't unduly split the value, but then the spaces are meaningful in the remote shell.

Using nested quotes naively does not solve the problem. For example ssh -i key 10.10.10.10 mkdir -p "'${DEST_PATH}/subdir1'" "'${DEST_PATH}/subdir2'" would work with file names with spaces, but file names containing ' would not work (and could be a remote command execution vulnerability if an adversary controls the value of DEST_PATH).

The easy way out is to run sshfs and treat everything as local paths. If you never need to pass file names around in ssh or rsync commands, you never need to worry about quoting those file names. However sshfs isn't always convenient to manage, and doesn't lend to efficient updates to large files with rsync.

If the easy way out doesn't work for you, you need to quote the value of DEST_PATH. That is, you need to change /path/with three spaces into something like '/path/with three spaces' or /path/with\ three\ \ spaces.

set_quoted () {
   raw="$1"
   quoted=
   while case "$raw" in *\'*) true;; *) false;; esac; do
     quoted="$quoted'\\''${raw%%\'*}"
     raw="${raw#*\'}"
   done
   quoted="'$quoted$raw'"
}
set_quoted "$DEST_PATH"; quoted_DEST_PATH="$quoted"
ssh -i key 10.10.10.10 mkdir -p "${quoted_DEST_PATH}/subdir1" "${quoted_DEST_PATH}/subdir2"
rsync "${SOURCE_PATH}" "$DEST_HOST:${quoted_DEST_PATH}/subdir1"

¹ _{Rsync has no problem with arbitrary file names in recursion or in local files. But it passes command line arguments that are a remote file name to a remote shell, which expands them.}

Note that that approach assumes the login shell of the user on the remote system (which is used to interpret the code made up of the concatenation of the arguments passed to ssh) is Bourne-like (with the exception of yash which has a further restriction that its code must be made a text) or rc-like. That wouldn't work reliably if the user's login shell was csh-like or fish-like. — Stéphane Chazelas, Jan 19 '21 at 20:14
rsync doesn't escape arguments for the shell, the local file names are not passed inside the remote shell code, but via a remote rsync -d's stdin in its own protocol. For the destination file/directory, rsync doesn't escape things, you have to do it yourself in the syntax of the remote shell, or better use -s/--protect-args for those to also be passed via rsync -d's stdin, avoiding interpretation by the user's shell again. — Stéphane Chazelas, Jan 19 '21 at 20:21
The set_quoted function puts quotes in the wrong places; the loop should use quoted="$quoted${raw%%\'*}'\\''". Also, if the local shell supports substitutions in variable expansions (i.e. bash, zsh, ksh, but not dash), you can replace the entire function with quoted_DEST_PATH=\'${DEST_PATH//\'/"'\\''"}\'. Note that due to parsing inconsistencies, this substitution may not work if used anywhere other than an unquoted right-hand-side of an assignment command. — Gordon Davisson, Jan 19 '21 at 23:30
Sorry, I meant rsync --server above instead of rsync -d, and remote instead of destination. — Stéphane Chazelas, Jan 20 '21 at 07:39
@GordonDavisson, About that see also: Escape a variable for use as content of another script — Stéphane Chazelas, Jan 20 '21 at 13:01

score 0 · Answer 3 · answered Jan 20 '21 at 19:34

If you have a GNU system, your printf supports %q, which makes the argument reusable as shell input.

So you can do

DEST_PATH_ESC="$(printf "%q" "$DEST_PATH")
ssh -i key 10.10.10.10 mkdir -p "${DEST_PATH_ESC}/subdir1" "${DEST_PATH_ESC}/subdir2"
rsync "${SOURCE_PATH}" "$DEST_HOST:${DEST_PATH_ESC}/subdir1" # not sure if you need it here

Let's test it:

a="some complicated thing with spaces
and such"
b=$(printf "%q" "$a")
echo "$b"

and we get

$'some complicated thing with spaces\nand such'

So that should be quite safe, not only for spaces, but also for newlines.

How would you gracefully handle this snippet to allow for spaces in directories?

3 Answers3