Why is `[` a shell builtin and `[[` a shell keyword?

Question

As far as I know, [[ is an enhanced version of [, but I am confused when I see [[ as a keyword and [ being shown as a builtin.

[root@server ~]# type [
[ is a shell builtin
[root@server ~]# type [[
[[ is a shell keyword

TLDP says

A builtin may be a synonym to a system command of the same name, but Bash reimplements it internally. For example, the Bash echo command is not the same as /bin/echo, although their behavior is almost identical.

and

A keyword is a reserved word, token or operator. Keywords have a special meaning to the shell, and indeed are the building blocks of the shell's syntax. As examples, for, while, do, and ! are keywords. Similar to a builtin, a keyword is hard-coded into Bash, but unlike a builtin, a keyword is not in itself a command, but a subunit of a command construct. [2]

Shouldn't that make both [ and [[ a keyword? Is there anything that I am missing here? Also, this link re-affirms that both [ and [[ should belong to the same kind.

Answer 1

The difference between [ and [[ is quite fundamental.

• [ is a command. Its arguments are processed just the way any other commands arguments are processed. For example, consider:

  [ -z $name ]
  

  The shell will expand $name and perform both word splitting and filename generation on the result, just as it would for any other command.

  As an example, the following will fail:

  $ name="here and there"
  $ [ -n $name ] && echo not empty
  bash: [: too many arguments
  

  To have this work correctly, quotes are necessary:

  $ [ -n "$name" ] && echo not empty
  not empty
  

• [[ is a shell keyword and its arguments are processed according to special rules. For example, consider:

  [[ -z $name ]]
  

  The shell will expand $name but, unlike any other command, it will perform neither word splitting nor filename generation on the result. For example, the following will succeed despite the spaces embedded in name:

  $ name="here and there"
  $ [[ -n $name ]] && echo not empty
  not empty
  

Summary

[ is a command and is subject to the same rules as all other commands that the shell executes.

Because [[ is a keyword, not a command, however, the shell treats it specially and it operates under very different rules.

Answer 2

In V7 Unix — where the Bourne shell made its debut — [ was called test, and it existed only as /bin/test. So, code you would write today as:

if [ "$foo" = "bar" ] ; then ...

you would have written instead as

if test "$foo" = "bar" ; then ...

This second notation still exists, and I find that it's more clear about what's going on: you are calling a command called test, which evaluates its arguments and returns an exit status code that if uses to decide what to do next. That command may be built into the shell, or it may be an external program.¹

[ as an alternative to test came later.² It may be a builtin synonym for test, but it is also provided as /bin/[ on modern systems for shells that do not have it as a builtin.

[ and test may be implemented using the same code. This is the case for /bin/[ and /bin/test on OS X, where these are hard links to the same executable.³ As a result, the implementation completely ignores the trailing ]: it doesn't require it if you call it as /bin/[, and it doesn't complain if you do provide it to /bin/test.⁴

None of that history affects [[, because there never was a primordial program called [[. It exists purely inside those shells that implement it as an extension to the POSIX shell.

Part of the distinction between "builtin" and "keyword" is due to this history. It also reflects the fact that the syntax rules for parsing [[ expressions is different, as pointed out in John1024's answer.⁵

---

Footnotes:

1. When you look at it that way, it makes it clear why you must put spaces around [ in shell scripts, unlike the way parentheses and brackets work in most other programming languages. If the shell's command parser allowed if["$x"..., it would also have to allow iftest"$x"...

2. It happened around 1980. /bin/[ doesn't exist in my copy of Ancient Unix V7 from 1979, nor does man test document it as an alias. In the corresponding man page entry I have in a pre-release copy of the System III manual from 1980, it is listed.

3. ls -i /bin/[ /bin/test

4. But don't count on this behavior. The Bash built-in version of [ does require the closing ], and its built-in test implementation will complain if you do provide it.

5. The builtin vs external command distinction may also matter for another reason: the two implementations may behave differently. This is the case for echo on many systems. Because there is only one implementation, no such distinction needs to be made for a keyword.

Answer 3

[ was originally just an external command, another name for /bin/test. But a few commands, such as [ and echo, are used so frequently in shell scripts that the shell implementors decided to copy the code directly into the shell itself, rather than have to run another process every time they're used. That turned these commands into "builtins", although you can still invoke the external program via its full path.

[[ came much later. Although the builtin is implemented internally within the shell, it's parsed just like external commands. As explained in John1024's answer, this means that unquoted variables will get word splitting done on them, and tokens like > and < are processed as I/O redirection. This made writing complex comparison expressions inconvenient. [[ was created as shell syntax, so that it could be parsed ideosyncratically. Within [[ variables don't get word splitting, < and > can be used as comparison operators, = can behave differently depending on whether the next parameter is quoted or not, etc. These are all conveniences that make [[ easier to use than the traditional [ command/builtin.

They couldn't simply recode [ as syntax like this because it would have been an incompatible change to millions of scripts. By using the new [[ syntax, which didn't previously exist, they could totally revamp the way it's used in an upward compatible way.

This is similar to the evolution that resulted in $((...)) syntax for arithmetic expressions, which has mostly replaced the traditional expr command.

Answer 4

The newer [[ in bash is an optimization of [.

The classic [ has one big drawback, when it's used frequently to do a trivial operation: it will spawn a new process every time:
(It creates a new address space just for comparing 0 and 1! Every time!)

I think a main point of the addition of [[ was making the evaluation of the expression inside [ not spawn an extra process. But how [ is working could not be changed - it would create lots of confusion and problems. So, the optimisation was implemented with a new name, in a more efficient way, namely a shell builtin command.
It became keyword in shell syntax as a side effect.

At the time [ was used first, it was the right way to do it with an external process.