What is (exactly) "list context" (and "string context")?

Question

I have seen several times the use of "list context" and "string context".

I know and understand the use of such descriptions in perl. They apply to $ and @.

However, when used in shell descriptions:

• two contexts in shell syntax: list context and string context.
• the absence of quotes (in list contexts)
• You always need quotes around variables in all list contexts
• leaving a variable unquoted in list context (as in echo $var)

They seem diffuse as a term that has not been defined anywhere or
at best, poorly documented.

There is no definition in POSIX for that, acording to google

Is this (from this) the gist of it ? :

In a nutshell, double quotes are necessary wherever a list of words or a pattern is expected. They are optional in contexts where a raw string is expected by the parser.

But it seems like a dificult term to use. How could we find "what the result should be" when "the result is needed" to know if it is a string or list context.

Or could it be preciselly and correctly defined?

Answer 1

There is no such concept in the standard shell language. There are no "contexts" only expansion steps.

Quotes are first identified in the tokenization which produces words. They glue words together so that abc"spaces here"xyz is one "word".

The important thing to understand is that quotes are preserved through the subsequent expansion steps, and the original quotes are distinguished from quotes that might arise out of expansions.

Parameters are expanded without regard for double quotes. Later, though, a field splitting process takes place which harkens back to the first tokenization. Once again, quotes prevent splitting and, once again, are preserved.

Pathname expansion ("globbing") takes place after this splitting. The preserved quotes prevent it: globbing operators are not recognized inside quotes.

Finally the quotes are removed by a late stage called "quote removal". Of course, only the original quotes!

POSIX does a good job of presenting the process in a way that is understandable; attempts to demystify it with extraneous concepts (that may be misleading) are only going to muddle the understanding.

People throwing around ad hoc concepts like "list context" should formalize their thinking to the point that it can provide a complete alternative specification for all of the processing, which is equivalent (produces the same results). And then, avoid mixing concepts between the parallel designs: use one explanation or the other. A "list context" or "string context" makes sense in a theory of shell expansion in which these are well defined, and the processing steps are organized around these concepts.

If I were to guess, then "list context" refers to the idea that the shell is working with a list of tokenized words such as the two-word list {foo} {abc" x "def}. The quotes are not part of the second word: its content is actually abc x def; they are semantic quotes which prevent the splitting on whitespace. Inside these quotes, we have "string context".

However, a possible implementation of these expansion steps is not to actually have quotes which are identified as the original quotes, but some sort of list data structure, so that {foo} {abc" x "def} is, say, a list of lists in which the quoted parts are identified as different kinds of nodes (and the quotes are gone). Using Lisp notation it could be:

(("foo") ;; one-element word
 ("abc" (:dq-str " x ") "def")) ;; three-element word

The nodes without a label are literal text, :dq-str is a double-quote region. Another type could be :sq-str for a single quoted item.

The expansion can walk this structure, and then do different things based on whether it's looking at a string object, a :dq-str expression or whatever. File expansion and field splitting would be suppressed within both :dq-str or :sq-str. But parameter expansion does take place within :dq-str. "Quote removal" would then correspond to a final pass which takes the pieces and catenates the strings, flattening the interior list structure and losing the type indicating symbols, resulting in:

("foo"
 "abc x def") ;; plain string list, usable as command arguments

Now here, note how in the second item we have ("abc" (:dq-str " x ") "def"). The first and last items are unwrapped: they are direct elements of the list and so we can say these are in the "list context". Whereas, the middle " x " is wrapped in a :dq-str expression, so that is "(double quoted) string context".

What "list" refers to in "list context" is anyone's guess without a clearly defined model such as this. Is it the master word list? Or a list of chunks representing one word?

Answer 2

Since most of the occurrences you reference are by me, I feel I have to give an answer here, though I will mostly paraphrase @Gilles.

I've been using list context vs scalar/non-list context (better than string context which can be confusing if not understood as non-list context) dozens of times since at least 2004 on usenet or unix.SE most of the time in articles discussing the impact of leaving expansions unquoted in Bourne-like shells. I don't remember anyone requesting clarification as to what I meant by that before (I do often try to give some examples of such contexts for illustration)

I've not used it in formal specification of any shell languages, that's just English text to help explain the shell behaviour to other persons.

That's not official terminology though that's obviously inspired from perl official (in the documentation) terminology. I can't tell if other persons have used those in the context of Unix shells before me (though it's very likely they did), but certainly people have since. I don't claim ownership of that.

list context (at least when I used it in the contexts I've used it) simply means contexts where the shell is expecting any number of elements. While scalar/non-list/string context would be where only one (or a single string/scalar if you want) is expected, like in perl. In most Bourne-like shells, those list contexts are:

• simple command arguments (as in echo elements)
• for i in elements
• array=(elements) (and variant with +=)

Some shells have more like:

• cmd < elements in zsh which does something similar to cat -- elements | cmd (as in nl < *.txt, nl < {foo,bar}.txt but nl < foo.txt < bar.txt).
• cmd > elements (and variants with >| >>...) in zsh which does something similar to cmd | tee -- elements
• elements() { code; } in zsh to define one ore more functions at once (or nothing if elements resolves to an empty list (though a literal () { echo x; } is an anonymous function)).
• compound=(foo=(elements) elements) or matrix=((elements) (elements)) and so on in ksh93.
• etc.

In those contexts, typically, globs are expanded and you need to quote your expansions if you don't want split+glob (or just empty-removal with zsh unless you enable the shwordsplit/globsubst sh-compatibility options) to be applied to them.

For instance, if you replace elements with *.txt in the examples above, *.txt will be expanded to the list of txt files in the current directory.

If you're looking for an equivalent in the POSIX specification, look for the contexts where globs are expanded. POSIX, in at least one instance refers to that as a context where field splitting will be performed (wording which was actually changed after I raised issues with the previous wording to the Austin group). Of course, that wording is not very useful to answer the questions about where field splitting is performed.

The scalar contexts would be the other contexts.

In

scalar=*.txt
case *.txt in...
[[ -f *.txt ]]

*.txt cannot be expanded because the shell is expecting just one string.

As a caveat/limitation, those terms don't cleanly capture what happens in cmd > * or cmd > ~(N)pattern or a=(); b=; c=(a b); d=*; IFS=:; e=a:b; cmd 1> "${a[@]}" 2> $b 3> "${c[@]}" 4> $d 5> $e in shells like bash/yash (when not in POSIX mode), ksh88 (using set -A instead of var=(...) syntax) or ksh93 (only when interactive with some), where it could be seen as another list context except that only a list with one element is expected (with splitting and globbing working differently for some).

Answer 3

The wording “list context” and “string context” comes from Perl, but similar concepts apply to the shell language. Note that these are similar concepts: the kinds of contexts and the consequences of the context type are different.

The word context is a technical term in programming language semantics. Its exact meaning is tied to a particular semantic formalization, which is beyond the scope of this answer. The cognitive meaning is the nature of the surroundings of a code snippet. For example, saying that the code snippet $foo has a different meaning in different contexts means that the behavior of a program containing $foo depends on the nature of what is around that occurrence of $foo in the program.

The semantics of the shell is rather complex. It doesn't fall neatly into traditional categories that you'll find in introductory textbooks on programming languages. The execution of a shell program can be broken into two phases (note that this is a way to present the semantics, it doesn't mean that a shell interpreter has to be broken up in this way):

1. A parsing stage turns a string (the content of the source file or of the argument to -c) into an abstract syntax tree. In the POSIX specification, this corresponds to steps 2 and 3 (token recognition and parsing). The POSIX specification defines grammar rules that describe the shape of the tree. Note that this isn't a context-free grammar — the presentation is based on the usual presentation of context-free grammars, but the annotations “apply rule N” make it a more complex mathematical object.

2. An execution stage performs some evaluation on the nodes of the tree, and calls external commands. In the POSIX specification, this corresponds to steps 4–7 (expansion, redirection, command execution and waiting).

Expansion is a process that applies to a particular type of node in the abstract syntax tree, which POSIX calls WORD and which is commonly called “word”. It can be divided into two groups.

1. The first group of expansion consists, in POSIX terminology, of tilde expansion (e.g. ~foo → /home/foo), parameter expansion (e.g. $foo → bar if the value of foo is bar), command substitution (e.g. $(foo) → bar if the output of the command foo is bar) and arithmetic expansion (e.g. $((2+2)) → 4). This first group of expansions is performed on every word, excluding characters that “quoted” by virtue of being inside single quotes or preceded by a backslash. The output of this group of expansions is approximately a string with annotations (I'll explain the approximation below).

2. The second group of expansion consists of field splitting and pathname expansion (commonly known as “filename generation” or “globbing”). This group of expansions turns an annotated string into a list of strings. This group of expansions is performed on a subset of the places where the first group is performed: it is not performed on parts of words that are quoted with double quotes, and it is not performed at all if the word is in certain positions in the abstract syntax tree. This is where list and string contexts come in: in certain contexts, i.e. for certain classes of positions in the abstract syntax tree, the second group of expansions is performed. These are list contexts, so named because the outcome of the expansion process is a list (of strings). In the other contexts, called string contexts, the second group of expansions is not performed, and the outcome of the expansion process is a single string.

POSIX describes quote removal as happening as the last expansion stage. This is one way to explain quoting, with all expansions before field splitting being defined as a transformation from a string to a string. For example, given the word '$foo'$bar\$qux, assuming that the value of the variable bar is value, parameter expansion turns this into '$foo'value\$qux and the other first-group expansions leave the string unchanged. Quote removal finally strips the quotes to get $foovalue$qux.

The presentation with quote removal requires performing quote matching at each stage. A presentation that's simpler to follow and implement, and gives the same end result, is to perform a dequoting stage that results in a list of parts. Each part is annotated to remember whether it was quoted. For example, '$foo'$bar\$qux dequotes to the following parts: quoted $foo, unquoted variable expansion of bar, quoted $, bare q, bare u, bare x. (Distinguishing between “quoted” and “bare” is necessary for things like identifying assignments and deciding whether to expand aliases.) Second-stage expansions only happen to unquoted parts in list context.

POSIX specifies whether the second group of expansions happen by explicitly listing the expansion stages. For example, “Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value”. A simpler way to phrase this is that only the first-group expansions happen, i.e. that assignment is a string context. There are only two context because there are only two sets of rules for performing expansions: either all of the first group is performed (string context), or both groups are performed in order (list context).

(Actually, to be complete, there is a third kind of context: case pattern context. In a case pattern, only first-group expansions are performed (like in string context), but a part of the second group of expansions is relevant — unquoted globbing characters are wildcards for the string matching.)

The definition of the language specifies which contexts are list contexts and which are string contexts. In principle, this could be arbitrary. However there is an intuition behind it: in places where the grammar expects a list of WORD tokens, second-group expansion is performed on these tokens, whereas in places where the grammar wants a single WORD, second-group expansion is not performed. A simple way to explain this is that where the grammar expects a list, it's a list context, and where the grammar expects a single string, it's a string context.