Bash: Escaping double quotes in $() command substitution

Question

I am having trouble wrapping my head around how command substitution works when part of the command's parameters come from a variable.

To illustrate, I'll just present a series of commands:

I'll first create a directory A B C in my tmp directory, in the new directory I'll create a new file abcfile:

user@desktop /tmp
$ pwd
/tmp

user@desktop /tmp
$ mkdir "A B C"

user@desktop /tmp
$ ls
'A B C'

user@desktop /tmp
$ touch "A B C"/abcfile

user@desktop /tmp
$ ls "A B C"/
abcfile

I'll new assign the value A B C to a variable $DIR and attempt to call ls to list the contents of this directory using various ways of calling the command

user@desktop /tmp
$ DIR="A B C"

With the variable unquoted, the command fails as expected

user@desktop /tmp
$ ls $DIR
ls: cannot access 'A': No such file or directory
ls: cannot access 'B': No such file or directory
ls: cannot access 'C': No such file or directory

With the variable quoted, it works as expected

user@desktop /tmp
$ ls "$DIR"
abcfile

If I want the output of the command as a string

user@desktop /tmp
$ echo $(ls "$DIR")
abcfile

This command works, but why does it work? According to shellcheck this is actually not correct, the variable should be outside of double quotes

user@desktop /tmp
$ echo "$(ls "$DIR")"
abcfile

If the inner variable is not double quoted, the command fails

user@desktop /tmp
$ echo "$(ls $DIR)"
ls: cannot access 'A': No such file or directory
ls: cannot access 'B': No such file or directory
ls: cannot access 'C': No such file or directory

n.b. that notwithstanding the syntax highlighting here, the only part of "$(ls "$DIR")" that isn't quoted is ls. I don't think the duplicate draws that to the fore, but it helps to understand that. — Michael Homer, Jun 26 '19 at 06:32

Kusalananda · Answer 1 · 2019-06-26T09:42:07.203

I'm assuming that you know that echo "$(somecommand)" could just be replaced by somecommand as echo is not needed for outputting the output of a command, and that you use this construct just as an example.

The command echo $(ls "$DIR") "works" because ls "$DIR" outputs abcfile which echo outputs.

Quoting the command substitution here will not make a difference in this case. The string abcfile does not need quoting, unless you change $IFS to include characters present in that filename (see why below).

However, consider

A B C/
|-- A*
`-- abcfile

Now:

$ echo "$(ls "$DIR")"
A*
abcfile

$ echo $(ls "$DIR")
A B C abcfile

You will notice that the last output expanded the filename A* as a filename globbing pattern, which matches the name of the directory A B C. You will also notice that we lost the newline that ls outputted.

The newline was lost because the shell did word-splitting on the unquoted output of ls, splitting it into words on spaces, tabs and newlines (the default contents of $IFS).

The name of the directory A B C was inserted because the words generated by the word splitting underwent filename generation (globbing).

If the name of the new file had been A* A*, the name of he directory would have been inserted twice:

A B C/
|-- A* A*
`-- abcfile

$ echo $(ls "$DIR")
A B C A B C abcfile

When is double-quoting necessary?

Hi Kusalananda,
I'm puzzled, is it really possible a filename with a "*" character in it?

Also, my question is also more with regards to how a variable is expanded within a double quoted string. In the context of a command, having the parameter value double quoted makes a difference (as the command will receive the parameter value as a single value, even if it may have space characters in it). — wshyang, Jun 26 '19 at 07:00
@wshyang You can have filenames with * in it, yes. You can also have newlines, tabs, and many other "strange" characters in filenames. You can have filenames with any character except for / (which is the path separator) and the \0 character (which is a string terminator in the C language). I don't understand your question about variables in double quoted strings. A variable within a double quoted string will be expanded, and the resulting value will be part of the double quoted string. It is not different if this string is then used by a command. — Kusalananda, Jun 26 '19 at 07:20

score 1 · Answer 2 · edited Jun 11 '20 at 14:16

It seems this is the main misunderstanding:

the variable should be outside of double quotes
user@desktop /tmp
$ echo "$(ls "$DIR")"
abcfile

The variable is not outside of double quotes. The shell considers quoting inside $() separately than quoting outside it. This means in echo "$(ls "$DIR")" the double-quotes are in fact nested:

echo "$(ls "$DIR")"
#          ^    ^     inner quotes
#    ^            ^   outer quotes

And because in general it's good to double quote variables and whole $() statements (unless you know you don't want to quote), you need each of these quotes.

Compare "quirk number 2" in this answer.

Side issue:

If I want the output of the command as a string
user@desktop /tmp
$ echo $(ls "$DIR")
abcfile

Output to stdout has no type, echo doesn't make it a string. In this particular case echo changes nothing but please see What is wrong with echo $(stuff)?

Bash: Escaping double quotes in $() command substitution

2 Answers2