Shell script output incorrectly splitting while being passed as an argument to script

Question

Let's say I have the following two shell scripts:

#!/bin/sh
#This script is named: args.sh

echo 1 "\"Two words\"" 3

, and:

#!/bin/sh
#This script is named: test.sh

echo "Argument 1: "$1
echo "Argument 2: "$2
echo "Argument 3: "$3

When I call the scripts as:

sh test.sh $(sh args.sh)

, I recieve:

Argument 1: 1
Argument 2: "Two
Argument 3: words"

When I expected instead to get:

Argument 1: 1
Argument 2: Two words
Argument 3: 3

Copying the output of sh args.sh and pasting it as the input of sh test.sh works just fine; so I assume this isn't actually what the shell is doing. I can achieve the desired/expected output by calling sh args.sh | xargs sh test.sh instead.

However, I'm wondering if there's an equivalent way of doing this without piping the output of the first script (args.sh) to xargs? I want the scripts to be called in the original order; the argument script to output the parameters to the second. I'm also looking for an explanation as to why this call doesn't work as expected?

score 4 · Answer 1 · answered Mar 21 '12 at 03:50

Part of the problem is that the string returned by args.sh is not parsed the same as a direct command, but only by the value of $IFS ($' \t\n'). Try turning on command tracing with set -x:

$ sh /tmp/test.sh $(sh /tmp/args.sh)
++ sh /tmp/args.sh
+ sh /tmp/test.sh 1 '"Two' 'words"' 3
Argument 1: 1
Argument 2: "Two
Argument 3: words"
$

Notice the line starting with a single +: there are four arguments, the '"Two' and 'words"' are two parsed as separate arguments. What you would want to do is change $IFS.

$ set -x
$ IFS='"'
+ IFS='"'
$ sh /tmp/test.sh $(sh /tmp/args.sh)
++ sh /tmp/args.sh
+ sh /tmp/test.sh '1 ' 'Two words' ' 3'
Argument 1: 1
Argument 2: Two words
Argument 3:  3
$

This won't work for every output. The best thing to do is to change the output of args.sh to separate the output by something different from a space, for example a comma or colon:

$ cat /tmp/args.sh
#!/bin/sh
#This script is named: args.sh

echo "1,Two words,3"
$ IFS=,
$ sh /tmp/test.sh $(sh /tmp/args.sh)
+ sh /tmp/args.sh
+ sh /tmp/test.sh 1 Two words 3
Argument 1: 1
Argument 2: Two words
Argument 3: 3
$

score 0 · Answer 2 · edited Apr 13 '17 at 12:36

When you leave a variable substitution $var or a command substitution $(cmd) unquoted, the result undergoes the following transformations:

Split the resulting string into words. The split happens at whitespace (sequences of space, tabs and newlines); this can be configured by setting IFS (see 1, 2).
Each of the resulting word is treated as a glob pattern, and if it matches some files, the word is replaced by the list of file names.

Note that the result is not a string but a list of strings. Furthermore, note that quote characters like " are not involved here; they are part of the shell source syntax, not of string expansion.

A general rule of shell programming is to always put double quotes around variable and command substitutions, unless you know why you have to leave them off. So in test.sh, write echo "Argument 1: $1". To pass arguments to test.sh, you're in trouble: you need to pass a list of words from args.sh to test.sh, but your chosen method involves a command substitution and that only provides the way for a simple string.

If you can guarantee that the arguments to be passed do not contain any newlines, and it is acceptable to change the invocation process slightly, you can set IFS to contain only a newline. Make sure that args.sh outputs exactly one file name per line with no stray quotes.

IFS='
'
test.sh $(args.sh)
unset IFS

If the arguments may contain arbitrary characters (presumably except null bytes, which you cannot pass as arguments), you'll need to perform some encoding. Any encoding will do; of course, it won't be the same as passing the arguments directly: that's not possible. For example, in args.sh (replace \t by an actual tab character if your shell doesn't support it):

for x; do
  printf '%s_\n' "$x" |
  sed -e 's/q/qq/g' -e 's/ /qs/g' -e 's/\t/qt/g' -e 's/$/qn/'
done

and in test.sh:

for arg; do
  decoded=$(printf '%s\n' "$arg" |
            sed -e 's/qs/ /g' -e 's/qt/\t/g' -e 's/qn/\n/g' -e 's/qq/q/g')
  decoded=${decoded%_qn}
  # process "$decoded"
done

You may prefer to change test.sh to accept a list of strings as input. If the strings don't contain newlines, invoke args.sh | test.sh and use this in args.sh (explanation):

while IFS= read -r line; do
  # process "$line"
done

Another method that prevents the need for quoting altogether is to invoke the second script from the first.

…
args.sh "$@"

Shell script output incorrectly splitting while being passed as an argument to script

2 Answers2