I don't really understand expansion and quoting on the shell (zsh/bash)

Question

I've read up on a few answers here in reference to quoting variables on the shell. Basically, here's what I'm running:

for f in "$(tmsu files)"; do echo "${f}"; done

this has the expected outcome: a list of files. However, I'm pretty sure they're coming out as a single string. After some reading I realized that this is a zsh-ism; it deals with splitting differently than most shells.

So I fired up bash and ran the same command. I was expecting it to split the list, but alas, I got exactly the same result. So now I'm really confused.

1. Why doesn't bash behave as expected?
2. How do I get zsh to split lines?

I've tried (zsh): for f in "$(tmsu files)"; do echo "${=f}"; done as well as for f in "$(=tmsu files)"; do echo "${f}"; done neither worked, so obviously I'm mis-understanding how to get zsh to split.

What makes this even worse is at some point I had this doing exactly what I wanted it to, and I can't remember how I did that.

Answer 1

for f in "$(tmsu files)"; do echo "${f}"; done

That's more like the wrong thing to do than a zsh-ism. The double-quotes tell the shell to keep the result of the expansion as a single string. In most cases that's what you want, e.g. if you have

filename="foo bar"
ls "$filename"

you want ls to get the filename as a single argument, and not as the two arguments foo and bar. (Those would be different filenames.) Now, the fact that you're using a command substitution there doesn't change that. Though you're right in that quoting doesn't solve your problem in this case, but that construct is awkward to begin with. (In a POSIX shell that is, zsh has tools of its own, of course).

Consider a list of files like:

hello.txt
some silly name with spaces.txt

Using "$(output filenames)" would give you that as one string, not two.
Using $(output filenames) would give you that as six strings, not two.

You could solve that by setting IFS to just the newline, so the latter would split on the newlines and give you the two filenames. But it's a nuisance, has a global effect, and you still need to disable globbing with set -f too in case the file name is funny * name.txt instead.

(This far it's still mostly the same with zsh, since it does split the results of an unquoted command substitution, even if it doesn't split a variable expansion. It doesn't do globbing on the result by default though.)

Now, the reason you can't see the difference is that echo prints all the arguments it gets, joined with single spaces. So echo foo bar and echo "foo bar" produce exactly the same output. Instead, use something like printf "<%s>\n" ..., which lets you see the argument borders more clearly:

$ printf "<%s>\n" foo bar
<foo>
<bar>
$ printf "<%s>\n" "foo bar"
<foo bar>

See also:

• Word Splitting on Greg's wiki
• Why you don't read lines with "for"
• How to loop over the lines of a file? (it says "lines of a file", but it doesn't matter if you have cat or whatever inside the command substitution), and as always:
• Why does my shell script choke on whitespace or other special characters?

Answer 2

However, I'm pretty sure they're coming out as a single string.

This is what you get when you use quotes. I can't speak to how zsh is supposed to do it, but the bash manual states as much (section "Word Splitting", under "EXPANSION"):

The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.

(emphasis added). Note that word splitting also does not occur with single quotes; however, substitution and expansion does not occur within them either.

If you want separate arguments out of the command substitution, you need to remove the quotes. However, if tmsu produces any results with spaces, those will also yield undesirable splits (since both spaces and newlines are considered word delimiters), making things a little more complicated. In bash, assuming tmsu gives one result per line, you can do this:

tmsu files |
    while IFS= read -r f
    do
        printf "<%s>\n" "$f"
    done

The read builtin reads a whole line of input. You can give it any number of names. All but the last are assigned based on the word delimiter (space by default), and the last is assigned the remainder of the line, so it's a good way to get around bash's annoying treatment of whitespace in expansion and substitution.

Answer 3

2. How do I get zsh to split lines?

With the f parameter expansion flag (details in man zshexpn. It says: Split the result of the expansion at newlines. This is a shorthand for ps:\n:.)

for f in ${(f)"$( tmsu files )"}; do
  print -ru2 -- $f
done

Sending output to stderr to leave stdout for printing pipeable data to (presumably there is more happening than just printing the output).

If your tool can produce output delimited by the ascii null character (aka NUL, \0), then zsh can split that output using the 0 flag (shorthand for ps:\0:) instead of f.

To preserve blank records, add the @ flag and " quotes (e.g. "${(@f)"$( command )"}"). Due to the nature of command substitution, it won't be possible to retrieve trailing newlines.

If you feel like using read in a while loop, something like the following should work in either zsh or bash:

while IFS= read -ru3 -d '' x || [[ $x ]]; do
  {
    printf >&2 '%s\n' "$x"
    z=$x
  } 3<&-
done 3< <(printf 'x\0y\0z')
printf 'z: %s\n' "$z"

No field splitting or backslash interpretation, the loop is not run in a subshell, stdin is still available, and last element is found despite the lack of a trailing delimiter.