Correctly quote array that is being passed indirectly via another command

Question

I need to pass an array of filenames to a command, preserving proper quoting. So far, so good. Unfortunately the command is actually a sub-command that is, in turn, invoked by another command. Concretely, the command is:

git filter-branch --index-filter \
    'git rm -rf --cached ‹file1› ‹file2›…' \
    HEAD

For simplicity, I’m going to replace this in the following by a simpler command that exhibits the same problem:

printf '%s\n' 'cmd file1 file2…'

Now I’ve got an array files=('a b' c). My desired result is that the above command prints in a single line, and individually quotes every token after cmd as necessary (e.g. when there’s a space).

It works if I manually expand and quote the file names:

$ printf '%s\n' 'cmd '\''a b'\'' c'
→ cmd 'a b' c

(Alternatively I could mix single and double quotes to achieve the same result.)

But it no longer works if I am trying to pass an array:

1. $ (set -x; printf '%s\n' "cmd '${files[@]}'")
   + printf '%s\n' 'cmd '\''a b' 'c'\'''
   → cmd 'a b
   c'
   

2. $ (set -x; printf '%s\n' 'cmd '\'"${files[@]}"\')
   + printf '%s\n' 'cmd '\''a b' 'c'\'''
   → cmd 'a b
   c'
   

3. $ (set -x; printf '%s\n' 'cmd '"${files[@]}")
   + printf '%s\n' 'cmd a b' c
   → cmd a b
   c
   

I’m not surprised (3) doesn’t work (and it’s only included for completeness). Based on the output of set -x, the shell correctly quotes the individual array elements in (1) and (2) and it even puts escaped quotes around the whole thing. But then it breaks apart the individually quoted items. Is there a way to prevent this?

---

Incidentally, Shellcheck (SC2145) suggests replacing the [@] part by [*] in the above. This obviously breaks for filenames with spaces.

Answer 1

1. Instead of an array, use set -- file1 file2 ... to fill the parameter list, then use bash parameter transformation with the Quote operator:

   set -- 'a "b' c "d 'e" "f 'g "'"h' ; (set -x; printf 'cmd %s\n' "${*@Q}")
   

   Output:

   + printf 'cmd %s\n' ''\''a "b'\'' '\''c'\'' '\''d '\''\'\'''\''e'\'' '\''f '\''\'\'''\''g "h'\'''
   cmd 'a "b' 'c' 'd '\''e' 'f '\''g "h'
   

   Or, if we remove the set -x; part, the output becomes:

   cmd 'a "b' 'c' 'd '\''e' 'f '\''g "h'
   

2. A comment from LL3 suggests a better way that doesn't require set -- ...:

   export x; n=(a "b 'c"); x="${n[@]@Q}"
   ( n=($x); printf 'cmd %s\n' "${n[*]}"; )
   

   Simple version:

   n=(a "b 'c"); echo "cmd ${n[@]@Q}"
   

   Output:

   cmd 'a' 'b '\''c'
   

3. Yet another method is to use bash parameter transformation with the Assignment operator, (which also needs an eval):

   export x;n=(a b 'c d');x="${n[@]@A}"; (eval "$x";printf '%s\n' "${n[@]}")
   

   Output showing what printf sees:

   a
   b
   c d
   

Answer 2

git filter-branch runs /bin/sh /usr/lib/git-core/git-filter-branch and that script evaluates the argument of --index-filter using eval.

So that argument is evaluated as /bin/sh code.

On most systems, /bin/sh will be more or less an interpreter of the POSIX sh language, though in a few like Solaris 10 and older, it could still be the ancient Bourne sh language instead.

When it comes to quoting syntax, it makes little difference though.

In any case, none of the ksh/bash/zsh extended quoting operators like $'...' can be used. What that means is that you can't use GNU/bash/zsh/ksh printf %q or mksh/bash ${var@Q} operator, or the xtrace tracing to generate the quoting as those resort to $'...' in some cases. They also use some forms of quoting that are not localisation-safe (like \).

One builtin quoting operator you could use is zsh's qq parameter expansion flag as it uses single quotes:

files=(foo 'a b c' $'a\nb\nc' --foo-- "a'b")
git filter-branch --index-filter "git rm -rf --cached -- ${${(@qq)files}}" HEAD

To see how zsh quotes those:

$ printf '<%s>\n' "${${(@qq)files}}"
<'foo' 'a b c' 'a
b
c' '--foo--' 'a'\''b'>

With bash/ksh/yash/zsh, you could do that same quoting using a function like:

shquote() {
  LC_ALL=C awk -v q=\' '
    BEGIN{
      for (i=1; i<ARGC; i++) {
        gsub(q, q "\\" q q, ARGV[i])
        printf "%s ", q ARGV[i] q
      }
      print ""
    }' "$@"
}

And then:

git filter-branch --index-filter "git rm -rf --cached -- $(shquote "${files[@]}")" HEAD

Answer 3

In Zsh there are several options for quoting. The best are the (q+) or (q-) expansion flags documented in zshall(1). These add fewer unnecessary characters:

$ cmd=(ssh localhost "echo hi > t")
$ newcmd=(sh -c "${${(q@)cmd}}"); echo "${${(q@)newcmd}}"
sh -c ssh\ localhost\ echo\ hi\ \>\ t
$ newcmd=(sh -c "${${(qq@)cmd}}"); echo "${${(qq@)newcmd}}"
'sh' '-c' ''''ssh''' '''localhost''' '''echo hi > t''''
$ newcmd=(sh -c "${${(q-@)cmd}}"); echo "${${(q-@)newcmd}}"
sh -c 'ssh localhost '''echo hi > t''
$ newcmd=(sh -c "${${(qqqq@)cmd}}"); echo "${${(qqqq@)newcmd}}"
$'sh' $'-c' $'$'ssh' $'localhost' $'echo hi > t''

As for the syntax of "${${(q@)cmd}}", the q (or qq, q- etc.) causes escaping or quoting to be applied. The @ causes this escaping to applied to each element of the array cmd. The outer ${...} seems to be equivalent to ${(j: :)...}, i.e. joining with spaces. The double quotes are needed so that the result is not split again.

Unfortunately all of the quoting mechanisms in Zsh and Bash are exponential in quote depth for some inputs.

Here is an example showing the growth rate for the various quote expansion operators (code below):

q: (1) 6; (2) 14; (3) 24; (4) 42; (5) 76; (6) 142; (7) 272; (8) 530; 
qq: (1) 5; (2) 15; (3) 43; (4) 125; (5) 369; (6) 1099; (7) 3287; (8) 9849; 
qqq: (1) 5; (2) 13; (3) 25; (4) 45; (5) 81; (6) 149; (7) 281; (8) 541; 
qqqq: (1) 6; (2) 14; (3) 24; (4) 39; (5) 64; (6) 109; (7) 194; (8) 359; 
q-: (1) 5; (2) 15; (3) 39; (4) 97; (5) 237; (6) 575; (7) 1391; (8) 3361; 
q+: (1) 6; (2) 16; (3) 40; (4) 98; (5) 238; (6) 576; (7) 1392; (8) 3362; 

Strangely qqqq grows the most slowly, even though it doesn't start to fall behind until the 4th level of nesting.

Tcl is a great language that has nested quote operators with linear growth properties (see item 6 under man tcl).

Here is the code for the experiments. I used $'\t' as the initial string because it gives different lengths for q+ and q-.

f (){
  flag=$1
  echo -n "$flag: "
  str=$'\t'
  for i in $(seq 1 10); do
    eval 'str=\"${${('$flag'@)str}}\"'
    N=$(echo -n $str | wc -c)
    echo -n "($i) $N; "
  done
  echo
}
f q
f qq
f qqq
f qqqq
f q-
f q+

Answer 4

$ foo=(1 2 '3 4' 4 5)
$ printf "'%s'\n" "${foo[@]}"
'1'
'2'
'3 4'
'4'
'5'
$ subcommand() { printf "'%s'\n" "$@"; }
$ subcommand "${foo[@]}"
'1'
'2'
'3 4'
'4'
'5'

So let's adapt this to your specific use-case:

git filter-branch --index-filter \
    'git rm -rf --cached file1 file2 […]' \
    HEAD

In your case we need to be a little more creative though, and break things up into smaller pieces.

git filter-branch --index-filter  \
    'git rm -rf --cached [MAGIC]' \
    HEAD

The file list we're creating is where you need the "magic" to happen. The rest is all static, yes? And since you're scripting this, you don't need it to be on three lines, which simplifies things:

git filter-branch --index-filter 'git rm -rf --cached [MAGIC]' HEAD

So:

prefix="git filter-branch --index-filter 'git rm -rf --cached "
postfix="' HEAD"
magic="$(printf '"%s" ' "${file[@]}"'

And then if we execute:

${prefix}${magic}${postfix}

Thereby, we have assembled your command - albeit using "s to enclose your filenames rather than 's, since the filter-branch command is in 's already.