1

I was trying to use unquoted strings expansions to pass two arguments to tar; the first is the command-line flag --exclude and the second contains a * character. In an attempt to avoid premature globbing, I tried quoting the *:

shopt -s nullglob
x="--exclude '*'"
echo tar $x

To my surprise, the '*' completely disappeared! Here was the output:

tar --exclude

I understand that I could switch from POSIX shell to Bash and use arrays to avoid this headache. But I still wonder: what the hell was going on with that POSIX shell snippet? Why did it delete the quoted globs? I would have expected that it would leave the quoted part alone, or go all the way and expand the glob. It did neither of those.

hugomg
  • 5,747
  • 4
  • 39
  • 54

2 Answers2

4

First of all, you are already using bash - or at least some non-POSIX options. (Neither shopt nor nullglob is POSIX.)

Now let's explain what happened.

x="--exclude '*'"
echo tar $x

The variable $x is word-split on whitespace, and the resulting parts given over for globbing. --exclude has no wildcard and is left verbatim. '*' has no match (unless you have a file name literally starting and ending with a single quote), so because it contains a wildcard and you've set nullglob it is removed. The result is

tar --exclude

Since you are already using bash you might as well do it properly,

x=('--exclude' '*')
echo tar "${x[@]}"

If you want a POSIX solution the best I can offer is to reuse the main argument list:

set -- '--exclude' '*'
echo tar "$@"
ilkkachu
  • 138,973
Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • Shouldn't single-quotes prevent globbing from happening to the * at all? I guess not. weird. – davolfman Feb 07 '24 at 19:26
  • @davolfman, no, the single quotes are just data here (when coming from a variable expansion, that is). It's the same as how a="foo && bar"; echo $a prints foo && bar, instead of using && as an operator. The result of unquoted expansion is word-split and used for globbing only, it doesn't get parsed for shell syntax in general. Note that the opposite would mean that something like s="ain't so"; echo $s would cause a syntax error from the unclosed quoted string. But yes, one could think of the way unquoted expansions work as weird. – ilkkachu Feb 07 '24 at 21:17
  • Something like x="--exclude=*"; echo tar "$x" would also work here, as long as $x only contains a single arg to tar. (note I changed the space in the middle to = so as not to need splitting) – ilkkachu Feb 07 '24 at 21:25
  • I was under a mistaken impression that single quotes prevented all expansion since it prevents variable expansion. In that case they would have prevented globbing in the echo. – davolfman Feb 07 '24 at 22:02
  • @davolfman yes and no. Single quotes prevent expansion, but here they were inside a double-quoted string and so not treated by the shell as anything other than a ' character – Chris Davies Feb 07 '24 at 22:49
  • @ChrisDavies So it's not the type of quotes, I've now checked that. So I guess quotes are handled before the other expansions and as a result the quotes created by the variable expansion are not processed as quotes and get included during filename expansion? – davolfman Feb 07 '24 at 23:27
  • @davolfman a='apple'; echo "1.a=$a" '2.a=$a' Now repeat the process with a="'apple'" and a='"apple"'. Does that help? – Chris Davies Feb 08 '24 at 10:55
1

For corner cases like this the order of shell expansions in the Bash Documentation is usually the cause of unexpected behavior. Posix has more-or-less the same behavior but stated less clearly.

Here's what's going on:

You've turn nullglob so globs that fail are empty.

Next you set a variable containing a literal asterisk, and single quotes.

You then echo the contents of the variable and the following things happen:

  1. The variable is expanded into text. That text contains a '*'
  2. The '*' is substituted for any file names present, in spite of the single quotes that you'd think would make it a string literal only. Nullglob turns this into an empty string
  3. The command is executed
ilkkachu
  • 138,973
davolfman
  • 602
  • Single-quote chars within a double-quoted string are NOT 'consumed'; try x="it's 'me'!";echo "$x" – dave_thompson_085 Feb 07 '24 at 05:53
  • Yeah this is much weirder and stupider than I thought then. It looks a lot like nullglob is implemented incorrectly. – davolfman Feb 07 '24 at 19:24
  • So single quotes prevent variable expansion but double quotes prevent filename expansion? That's unpleasant – davolfman Feb 07 '24 at 19:35
  • What makes you think it's incorrect? A value of '*' in a variable (not part of the command input) is a glob for filenames that begin and end with singlequote characters; since there (almost certainly) are no such filenames, there is no match and nullglob removes it. Single quotes in command input not within other quotes disable all expansions; double quotes in command input not within other quotes disable brace and tilde expansions, wordsplitting, and globbing (and redirection and process substition) but allow parameter/variable and arithmetic expansions and command substitution. ... – dave_thompson_085 Feb 08 '24 at 03:21
  • ... Either kind of quote within the other kind of quotes or in a variable value is just data. (And in doublequotes backslash+doublequote is one doublequote as data.) – dave_thompson_085 Feb 08 '24 at 03:27