1

I see that in bash the command

echo $(((i=18)))

prints 18. This makes me understand that $(((i=18))) is interpreted as an arithmetic expansion (with the variable i being initialized inside the construct). Though, one could also think of a command substitution

$(command)

with

((i=18))

being the command. As a matter of fact, it looks command substitutions come before arithmetic expansions (Learning the bash Shell, O'Reilly 2005, p. 181). Therefore the result is not what one should expect. How do you explain this?

diciotto
  • 89
  • 1
    In this case the inner most parentheses are just used as arithmetic parentheses and since there is nothing outside of them they do nothing. $(((i=18))) is equivalent to $((i=18)). But now imagine $(((10-9)*2)) vs $((10-9*2)) – jesse_b Apr 13 '21 at 16:16
  • Yes, I already wrote in the beginning of my question. But you did not explain why command substitution (which should come first according to the precedence scheme) is not executed (while arithmetic expansion is instead) – diciotto Apr 13 '21 at 16:27
  • Who's precedence scheme? Maybe you shouldn't take books written by third parties as lore? Also I don't know what the book says on page 181 because I don't have the book so it's hard to reference what you are referring to. – jesse_b Apr 13 '21 at 16:33
  • Also the precedence wouldn't matter since $(( is arithmetic expansion and not command substitution – jesse_b Apr 13 '21 at 16:41
  • Well, then for the expansion order you can refer to this [link] (https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html). As you can see, also here command substitution comes before arithmetic expansion. I agree with you that arithmetic expansion is in fact executed here. Only that does not match what expected from those manuals. Also, I am not convinced about what operations are performed in the expansion. I used to think it was an initialization but in fact it is not. Try for example echo $(( 19 + i=9 )). It gives 9. Can you please explain why? P.S. ((i=18)) is a command – diciotto Apr 13 '21 at 16:55
  • Again you are referencing something other than the GNU bash manual or the official POSIX documentation. Anything it says in contradiction to the official documentation should be ignored. I think the confusion in these documents may come from the section just above the one xhienne posted in his answer where they list all the types of expansions (out of order). But either way that is assuming the shell source code is too dumb to detect whether the $ is followed by 1 or 2 opening parentheses. It very well may be but that is really easy logic to implement. – jesse_b Apr 13 '21 at 17:00
  • You are right, that's exactly what I was about to add. The section before that text lists the expansions in a different order. And that's what confused the two sources (and me from them) – diciotto Apr 13 '21 at 17:06
  • Quoting from the LDP document you referenced, "process substitution is performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion." – Chris Davies Apr 13 '21 at 19:06

2 Answers2

2

This behavior is consistent with what is stated in Bash's CHANGES file:

$((...)) is always parsed as an arithmetic expansion first, instead of as a potential nested command substitution, as Posix requires.

Indeed, POSIX acknowledges that there is an ambiguity and that arithmetic expansion must prevail:

The syntax of the shell command language has an ambiguity for expansions beginning with "$((", which can introduce an arithmetic expansion or a command substitution that starts with a subshell. Arithmetic expansion has precedence; that is, the shell shall first determine whether it can parse the expansion as an arithmetic expansion and shall only parse the expansion as a command substitution if it determines that it cannot parse the expansion as an arithmetic expansion.

Now some words on your sentence "it looks command substitutions come before arithmetic expansions": no, POSIX specifies that word expansion occurs from the beginning to the end, i.e. from left to right, in the order they are met:

Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end.

The "Token Recognition" section gives some further precision:

If the current character is an unquoted '$' or '`', the shell shall identify the start of any candidates for parameter expansion, command substitution, or arithmetic expansion from their introductory unquoted character sequences: '$' or "${", "$(" or '`', and "$((", respectively. The shell shall read sufficient input to determine the end of the unit to be expanded.

xhienne
  • 17,793
  • 2
  • 53
  • 69
  • Very interesting, thanks. Then it turns out two sources are flatly wrong! Apart from that, can you explain why echo $(( 19 + i=9 )) gives 9? – diciotto Apr 13 '21 at 17:01
  • your $(( 19 + i=9 )) command fails for me btw. "attempted assignment to non-variable". You can do $((19+(i=9))) and that gives 28 – jesse_b Apr 13 '21 at 17:02
  • Strange. In my bash (version 3.2.57(1)) echo $(( 19 + i=9 )) gives 9. – diciotto Apr 13 '21 at 17:08
  • @diciotto: I can reproduce on v3.2 but since it has apparently been fixed in later versions it is safe to assume that is indeed undesired behavior although it's also just a bad equation. You are saying 19+i and then =9 due to the order of operations. – jesse_b Apr 13 '21 at 17:32
  • 1
    @xhienne, I think you're misreading that quote from Bash's manual. Note the semicolons after brace expansion and before and after word splitting. That combined with the note about left-to-right evaluation seems to indicate that the order is 1) brace expansion; then 2) parameter/variable/expansions and command substitutions, all in left-to-right order; then 3) word splitting; then 4) pathname expansion. – ilkkachu Apr 13 '21 at 17:45
  • @xhienne yes, macOS 10.15.7. I was aware of the nonsensical nature of $(( 19 + i=9 )), only I was trying to explain the result. Now, as jesse_b seems to state, it simply means nothing, it's a bug. I am pleased with the outcome. Thanks – diciotto Apr 13 '21 at 17:51
  • @diciotto, that $(( 19 + i = 9 )) is a curious one, actually. The + should have precedence over the =, so it should parse as (19 + i) = 9. But actually putting the parenthesis there turns it into an error. With the parenthesis on the right, $(( 19 + (i = 9))) works, but of course has a different meaning (it returns 28). – ilkkachu Apr 13 '21 at 18:01
  • @xhienne, about the ordering of the expansions, note that even with order it's written in the manual, the result of a parameter expansion doesn't get further processed as a command substitution or arithmetic. E.g. foo='$(echo hi)'; echo $foo prints $(echo hi), and not hi. But brace expansion is a different step and it can produce command substitutions and variable expansions... E.g. echo $(echo foo >&2){a,b} prints foo to stderr twice (and a b to stdout), and echo $foo{a,b} prints whatever the variables fooa and foob contain. Unlike what Ksh and Zsh do, of course. – ilkkachu Apr 13 '21 at 18:11
  • @xhienne parameter and variable expansion are the same. The semi colons are used to separate the overall order and within some sections there are multiple operations which are separated by commas. The reason there is no oxford comma between parameter and variable expansion is intentional. – jesse_b Apr 13 '21 at 18:26
  • @jesse_b Oops, you are right, this is a whole, forget that part of my comment. – xhienne Apr 13 '21 at 18:28
  • @ilkkachu the result of a parameter expansion doesn't get further processed as a command substitution or arithmetic. Sorry, I may miss something but I don't get your point at all. I see nothing in the Bash manual I quote that contradicts the behavior you are observing (brace expansion comes first, quote removal comes last, hence your results) – xhienne Apr 13 '21 at 18:34
  • 1
    @xhienne, The two are the same, like Jesse said. The way POSIX puts it: "A variable is a parameter denoted by a name." (Of course, "name" also has a specific meaning there.) About how clear that is, well, I suppose you could suggest to the maintainer a better way of phrasing it. But, for all I can tell and experiment, that first statement saying "command substitution comes after arithmetic expansions" seems beside the point, misleading or outright wrong. That's what I was commenting about. – ilkkachu Apr 13 '21 at 18:34
  • @xhienne, right. If it were the case that parameter expansion would be processed first, and then command substitution after, it would mean that echo $foo would be first expanded to echo $(echo hi) (by parameter expansion), which would then be expanded to echo hi (by command substitution). (Assuming the assignment foo='$(echo hi)' beforehand, of course.) But that's not what happens. – ilkkachu Apr 13 '21 at 18:36
  • @ilkkachu Sorry, I still don't understand. I think you are referring to your foo='$(echo hi)'; echo $foo. Are you saying that the expansion of two unrelated commands (command1; command2) occurs at the same time??? That is not the case. The section about expansion that I quoted from the Bash manual is about the expansion of a simple command. First foo='$(echo hi) is expanded, then it is executed, then echo $foo is expanded, then it is executed. – xhienne Apr 13 '21 at 18:56
  • Let us continue this discussion in chat. – ilkkachu Apr 13 '21 at 19:23
2

Wrt. $(((i=18))), you're right, it could be either $(( (i=18) )), or $( ((i=18)) ), both are valid. A somewhat common (and definitely simple) way to interpret ambiguous cases like this is recognize the longest valid operator. That would mean this would interpreted as $(( followed by (.

That's what happens with e.g. <<( in Bash and Ksh: it's << followed by (, and not < followed by <(. Even though the first interpretation is a syntax error while the second would be valid! The user must add a space to help the shell figure it out. In the same way, i+++a is i++ + a and not i + ++a. Similar things exist in other languages, too.

But with $(((, that's not exactly the whole truth. Some shells do look further, some don't. Consider e.g. this:

echo $((echo hi); (echo ho))

If interpreted as a command substitution, it's valid, and prints hi ho. But greedily recognizing the $(( would have it interpreted as an arithmetic expansion, and as such, it's completely bogus.

Apart from Dash and Busybox, all shells I tried recognize the valid command substitution there. Putting a space between the first two parenthesis makes it unambigous.

As a matter of fact, it looks command substitutions come before arithmetic expansions

No, they happen at the same point of processing. To see that, create a command substitution that expands to what would be a valid arithmetic expansion. E.g. echo '$((1+2))' prints $((1+2)); so $( echo '$((1+2))' ) expands to $((1+2)). But that's not further processed within the same command.

This,

echo $( echo '$((1+2))' )

outputs $((1+2)) and not 3.

Of course an arithmetic expansion can only produce numbers, so the opposite order can't be tested. But similar experiments can be made between variable/parameter expansion and command substitution, and in none of the cases, the results of one expansion expand further.

Brace expansion, on the other hand is different.

Bash processes it before variable expansions:

$ bash -c 'v=000 va=123 vb=456; echo $v{a,b}; n=1 m=4; echo {$n..$m}'
123 456
{1..4}

while Ksh does the opposite:

$ ksh -c 'v=000 va=123 vb=456; echo $v{a,b}; n=1 m=4; echo {$n..$m}'
000a 000b
1 2 3 4

(In Ksh, even a="{1.."; b="4}"; echo $a$b expands the brace, and outputs 1 2 3 4. Zsh is again the sane one here, it expands variables first, but doesn't let expanded braces trigger further expansion. Zsh also recognizes <<( as < <(.)

And then of course, there's word splitting and filename generation, which happen after all others, but only for results of unquoted expansions.

ilkkachu
  • 138,973