How do you expand a variable in a heredoc as one argument when its value has spaces?

Question

I've created a script to upload via lftp:

#! /usr/bin/bash
set -xe
if [ -n "$1" ]; then
...
else
  source="."
target=name of current local folder
target="${PWD##*/}"/
  cmd="mirror --reverse --continue --parallel=5 "$source" "$target""
fi
lftp -u $user,$pass $host << EOF
set log:enabled true
eval "$cmd"
quit
EOF

If the current directory does not contain spaces, this works correctly. The problem occurs when the current directory does have spaces. Here is what the debug information shows:

➜  upload.sh
+ '[' -n '' ']'
+ source=.
+ target='Two Words/'
+ cmd='mirror --reverse --continue --parallel=5 . Two Words/'

As a result of this issue, the script uploads to directory named Two instead of Two Words.

I'm not sure how to fix this problem, especially because I don't know which line is where I went wrong: cmd=...? eval "$cmd"? Neither? Both?

Regardless, I was expecting that debug output to look like this: + cmd='mirror --reverse --continue --parallel=5 "." "Two Words/"' (notice the double quotes) and I'm not sure why it didn't.

This question is similar to a lot of others I researched to get this far. What makes this different/hard to Google is that the expansion is happening inside of a "here string". For all I know that's irrelevant, but for all I know it could be critically unique, too.

---

Comments suggested I store everything into an array. Here's my attempt to do so:

#! /usr/bin/bash
set -xe
if [ -n "$1" ]; then
  cmd=(mput -c -P 5 "$1")
  if [ -n "$2" ]; then
    cmd=(mkdir -p "$2"
        mput -c -P 5 -O "$2" "$1") #*
  fi
else
  source="."
target=name of current local folder
target="${PWD##*/}"/
  cmd=(mirror --reverse --continue --parallel=5 "$source" "$target")
fi
lftp -u $user,$pass $host << EOF
"${cmd[@]}"
quit
EOF

(* I can't figure out how to have multilines in a subshell. That's why I put actual newlines in there, but I'm not sure if that's the correct thing to do. Every result I google has to do with command substitution, not subshells.)

When I run this, it gives me this error: Unknown command 'mirror --reverse --continue --parallel=5 . Two Words/'. If I copy and paste that into my shell, it gives me a different error, so I'm not understanding what's going on.

Calling lftp -u $user,$pass $host -e "${cmd[@]}" didn't work either. That gives this error:

open: unrecognized option '--reverse'
Usage: lftp [-e cmd] [-p port] [-u user[,pass]] <host|url>

running lftp ... -e "$("${cmd[@]}")" gave this error:

open: option requires an argument -- 'e'
Usage: lftp [-e cmd] [-p port] [-u user[,pass]] <host|url>

As you can see, at this point, I'm just trying random stuff and hoping it'll provide insight. Not a good strategy.

Answer 1

You'd need to quote those arguments (or at least escape those spaces or other characters special in lftp's language) for lftp's mirror command in the syntax of the lftp language.

lftp supports a simple form of quoting. Whilst like in most shells it can be with single quotes, double quotes and backslashes, the syntax is different. You can experiment by using lftp's echo command.

lftp :~> echo \'
'
lftp :~> echo \a
\a
lftp :~> echo \
> a
a

\ is an escaping operator but only for some characters. When followed by newline, it's a line continuation, it doesn't escape the newline.

lftp :~> echo "a\'"
a\'
lftp :~> echo "a\"b"
a"b
lftp :~> echo "a\$b"
a\$b

Inside double quotes, the list of characters it escapes is further reduced. It's possible to have a litteral " inside "..." with \".

lftp :~> echo 'a\'b'
a'b
lftp :~> echo 'a\\b'
a\b

Same inside single quotes. '...' are not strong quotes there like they are in sh-like shells.

lftp :~> echo $'a\nb'
$a\nb

The ksh93-style $'...' form of quoting is not supported.

lftp :~> echo 'foo
foo
lftp :~> echo 'a\
b'
ab

Quotes don't have to be paired. It doesn't seem like it's possible to include newline in a command argument whether it's inside quotes or not.

From looking at the code, the characters that need to be escaped inside '...' or "..." are only backslash and the corresponding quote character. lftp works at byte level, it doesn't decode sequences of bytes into characters.

So here, the best approach for a string to be quoted properly in the syntax of lftp is to escape every ' and \ (actually 0x5c bytes, even those that are present in the encoding of other multibyte characters) in it with \ and wrap it in '...'. Or the same with "...".

Here, using eval here is unnecessary and adds more complication as you get two levels of quoting to manage.

#! /usr/bin/bash -
set -xe
lftp_quote() {
  local LC_ALL=C
  local -n var
  for var do
    if [[ $var = $'\n' ]]; then
      printf >&2 '%s\n' ""$var" contains newline characters. That's not supported by lftp"
      exit 1
    fi
    var=${var//''/'\'}
    var=${var//"'"/"'"}
    var="'$var'"
  done
}
if (( $# > 0 )); then
  ...
else
  source="."
target=name of current local folder
target="${PWD##*/}"/
  lftp_quote source target
  cmd="mirror --reverse --continue --parallel=5 -- $source $target"
fi
lftp_quote user pass host
lftp << EOF
set log:enabled true
open --user $user --password $pass -- $host && $cmd
EOF

(here also avoiding passing the password on the command line as that is public)

Now, while we may have managed to pass those strings as-is to lftp's mirror command, whether lftp itself will properly escape those file names in commands in the FTP protocol is another matter. FTP is notorious for being unreliable on that front with different servers behaving differently. Note that lftp supports many other file transfer protocoles besides FTP some of which are known to be more reliable.

Answer 2

I'm not sure how to fix this problem, especially because I don't know which line is where I went wrong: cmd=...? eval "$cmd"? Neither? Both?

Regardless, I was expecting that debug output to look like this: + cmd='mirror --reverse --continue --parallel=5 "." "Two Words/"' (notice the double quotes) and I'm not sure why it didn't.

What you had there was:

cmd="mirror --reverse --continue --parallel=5 "$source" "$target""

The first " (at cmd=") starts a quoted string, the second (at "$source) ends it, same for the next ones. The resulting string doesn't have any quotes in it. You'd need to use

cmd="mirror ... \"$source\" \"$target\""

to add double quotes in the string, or if single quotes work for lftp, you could use

cmd="mirror ... '$source' '$target'"

---

cmd=(mirror --reverse --continue --parallel=5 "$source" "$target")
lftp -u $user,$pass $host << EOF
"${cmd[@]}"

When I run this, it gives me this error: Unknown command 'mirror --reverse --continue --parallel=5 . Two Words/'.

Well. The "${array[@]}" expansion is useful in the shell in that it expands the array elements to multiple distinct shell words (strings), keeping e.g. filenames with whitespace intact. But here, we're inside a here-doc, so it's not possible to produce multiple words/strings: the whole here-doc is just one string. So it just joins the array elements with spaces and expands to that. The quotes there stay, as they do inside a heredoc.

The command lftp gets is

"mirror --reverse --continue ..."

Because of the quotes, it looks at that as a single command, compare e.g.

lftp :~> "echo foo"
Unknown command `echo foo'.
lftp :~> echo foo
foo

The question title also asks about "[expanding] a variable in a heredoc as one argument", but that's not the issue, since it's impossible to do otherwise (and one might say that the concept of an "argument" doesn't exist in a here-doc anyway). No, your problem is getting the quoting right for lftp.

If I copy and paste that into my shell, it gives me a different error, so I'm not understanding what's going on.

Probably something like bash: mirror: command not found? That mirror there is a command for lftp, so I'm not sure it'd be useful to paste it to the shell's command line.

I understand the suggestion of using an array for storing a shell command, there are some very good reasons for it. But you don't have a shell command here, so that's unnecessary.

Calling lftp -u $user,$pass $host -e "${cmd[@]}" didn't work either. That gives this error:

open: unrecognized option '--reverse'
Usage: lftp [-e cmd] [-p port] [-u user[,pass]] <host|url>

Here, you're passing the contents of that array on lftp's command line. Because you used [@], the array elements get passed as distinct arguments, so the command is the same as

lftp -u USER,PASSWORD HOSTNAME -e mirror --reverse --continue --parallel=5 SOURCE TARGET

(where $user, $host, $source and $target got expanded too, of course.)

The -e option takes a command as an argument, and here, that command is just mirror. The following one, --reverse is taken as another command-line option, and lftp doesn't recognize that. Here, you'd want to use "${cmd[*]}" instead, to avoid having the array elements as distinct args, since lftp's -e option can only take one string anyway.

On the other hand, since it can take only a single string anyway, using the array is kinda useless to begin with.

running lftp ... -e "$("${cmd[@]}")" gave this error:

The $(...) there is a command substitution, it'd run the insides as a command in the shell. I'm not sure where you pulled that from, but try e.g.

$ cmd=(date)
$ lftp ...  -e "$("${cmd[@]}")"

(and you probably should be happy it wasn't rm something in there.)

---

Stéphane's answer gave you a solution that handles quoting most special characters too, but if you want it simpler, and you're happy to assume you only have whitespace characters to worry about, I think this should work:

#! /usr/bin/bash
set -xe
source="."
target="Two Words/"
cmd="mirror --reverse --continue --parallel=5 '$source' '$target'"
user=foo
pass=secret
lftp -u "$user,$pass" http://localhost << EOF
set log:enabled true
$cmd
quit
EOF

We don't need an array, since it ends up as just a single string anyway. We do need to have quotes in the resulting string around the filenames; I used single quotes there inside $cmd and since this is the lazy solution, I didn't care if the filenames contain quotes too. You could instead use double-quotes with "... \"$source\"...". The here-doc here ends up as:

set log:enabled true
mirror --reverse --continue --parallel=5 '.' 'Two Words/'
quit

which looks like it would be a sensible set of command to pass to lftp.

Of course it'd be relatively simple to just test the filenames don't have quotes, before trying and seeing lftp croak:

case $target in
    *[\'\"]*) echo "ugh, can't have quotes in filename!"; exit 1;;
esac

or more strictly, listing the allowed characters:

case $target in
    *[![:alnum:]._" "/]*) echo "ugh, disallowed characters in filename!"; exit 1;;
esac