How does storing the regular expression in a shell variable avoid problems with quoting characters that are special to the shell?

Question

From Bash Manual

Storing the regular expression in a shell variable is often a useful way to avoid problems with quoting characters that are special to the shell. It is sometimes difficult to specify a regular expression literally without using quotes, or to keep track of the quoting used by regular expressions while paying attention to the shell’s quote removal. Using a shell variable to store the pattern decreases these problems. For example, the following are equivalent:

pattern='[[:space:]]*(a)?b'
[[ $line =~ $pattern ]]

and

[[ $line =~ [[:space:]]*(a)?b ]]

If you want to match a character that’s special to the regular expression grammar, it has to be quoted to remove its special meaning. This means that in the pattern xxx.txt, the . matches any character in the string (its usual regular expression meaning), but in the pattern "xxx.txt" it can only match a literal .. Shell programmers should take special care with backslashes, since back-slashes are used both by the shell and regular expressions to remove the special meaning from the following character. The following two sets of commands are not equivalent:

pattern='\.'

[[ . =~ $pattern ]]
[[ . =~ \. ]]

[[ . =~ "$pattern" ]]
[[ . =~ '\.' ]]

The first two matches will succeed, but the second two will not, because in the second two the backslash will be part of the pattern to be matched. In the first two examples, the backslash removes the special meaning from ., so the literal . matches. If the string in the first examples were anything other than ., say a, the pattern would not match, because the quoted . in the pattern loses its special meaning of matching any single character.

How is storing the regular expression in a shell variable a useful way to avoid problems with quoting characters that are special to the shell?

The given examples don't seem to explain that. In the given examples, the regex literals in one method and the values of the shell variable pattern in the other method are the same.

Thanks.

Answer 1

[[ ... ]] tokenisation clashes with regular expressions (more on that in my answer to your follow-up question) and \ is overloaded as a shell quoting operator and a regexp operator (with some interference between the two in bash), and even when there's no apparent reason for a clash, the behaviour can be surprising. Rules can be confusing.

Who can tell what these will do without trying it (on all possible input) with any given version of bash?

[[ $a = a|b ]]
[[ $a =~ a|b ]]
[[ $a =~ a&b ]]
[[ $a =~ (a|b) ]]
[[ $a =~ ([)}]*) ]]
[[ $a =~ [/\(] ]]
[[ $a =~ \s+ ]]
[[ $a =~ ( ) ]]
[[ $a =~ [ ] ]]
[[ $a =~ ([ ]) ]]

You can't quote the regexps, because if you do, since bash 3.2 and if bash 3.1 compatibility has not been enabled, quoting the regexps removes the special meaning of RE operator. For instance,

[[ $a =~ 'a|b' ]]

Matches if $a contains a litteral a|b only.

Storing the regexp in a variable avoids all those problems and also makes the code compatible to ksh93 and zsh (provided you limit yourself to POSIX EREs):

regexp='a|b'
[[ $a =~ $regexp ]] # $regexp should *not* be quoted.

There's no ambiguity in the parsing/tokenising of that shell command, and the regexp that is used is the one stored in the variable without any transformation.

Answer 2

The only way to match an explicit string is to quote it:

[[ $var =~ 'quux' ]]

Even if the string contains special characters (special to the shell^[a])
without the shell expanding or interpreting them^[b]:

$ var='^abcd'
$ [[ $var =~ '^ab' ]] && echo yes || echo no
yes

If we need to actually allow (shell) special characters and allow the shell to interpret them as a regular expression they should be un-quoted.

$ var='abcd'
$ [[ $var =~ ^ab ]] && echo yes || echo no
yes

But unquoted strings create new problems, like with spaces:

$ var='ab cd'
$ [[ $var =~ ^ab cd ]] && echo yes || echo no
bash: syntax error in conditional expression
bash: syntax error near `cd'

To solve it, we need to still quote special characters:

$ var='ab cd'
$ [[ $var =~ ^"ab cd" ]] && echo yes || echo no
yes

$ [[ $var =~ ^ab\ cd ]] && echo yes || echo no
yes

Other examples:

[[ "a b"  =~  ^a\ b$ ]] && echo yes
[[ "a|b"  =~  ^a\|b$ ]] && echo yes
[[ "a&b"  =~  ^a\&b$ ]] && echo yes

Storing the regexp inside a variable avoids all those quoting problems.

$ regex='^a b$'
$ [[ "a b" =~ $regex ]] && echo yes
yes

---

^[a] List of shell special characters (| & ; ( ) < > space tab newline).

^[b] This is true since bash version bash-3.2-alpha (under "3. New Features in Bash" heading):

f. Quoting the string argument to the [[ command's =~ operator now forces string matching, as with the other pattern-matching operators.

---

Copy of extended description from bash FAQ:

E14) Why does quoting the pattern argument to the regular expression matching conditional operator (=~) cause regexp matching to stop working?

In versions of bash prior to bash-3.2, the effect of quoting the regular expression argument to the [[ command's =~ operator was not specified. The practical effect was that double-quoting the pattern argument required backslashes to quote special pattern characters, which interfered with the backslash processing performed by double-quoted word expansion and was inconsistent with how the == shell pattern matching operator treated quoted characters.

In bash-3.2, the shell was changed to internally quote characters in single- and double-quoted string arguments to the =~ operator, which suppresses the special meaning of the characters special to regular expression processing ('.', '[', '\', '(', ')', '*', '+', '?', '{', '|', '^', and '$') and forces them to be matched literally. This is consistent with how the `==' pattern matching operator treats quoted portions of its pattern argument.

Since the treatment of quoted string arguments was changed, several issues have arisen, chief among them the problem of white space in pattern arguments and the differing treatment of quoted strings between bash-3.1 and bash-3.2. Both problems may be solved by using a shell variable to hold the pattern. Since word splitting is not performed when expanding shell variables in all operands of the [[ command, this allows users to quote patterns as they wish when assigning the variable, then expand the values to a single string that may contain whitespace. The first problem may be solved by using backslashes or any other quoting mechanism to escape the white space in the patterns.

Related questions:

Using a variable in a regex