Precedence of subshells in relation to redirection?

Question

Shell: GNU BASH

I've been using the following command for a while now:

echo "$(fmt -w 50 < foo.txt)" > foo.txt

Without any issues. However, when I try:

fmt -w 50 < foo.txt > foo.txt

The file foo.txt gets turned into a blank file. I assume that this is because of something to do with precedence of redirection operators and subshells being parsed by the shell.

Searching the GNU BASH manual hasn't been very fruitful. Could someone please tell me if I missed something in the manual? I would really like to understand the reason for this behaviour so I can make further use of it and know about any cases where it might lead to trouble.

score 2 · Accepted Answer · answered Jun 24 '19 at 16:29

From the Bash manual, 3.7.1 Simple Command Expansion:

When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.

The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.

The words that are not variable assignments or redirections are expanded (see Shell Expansions). If any words remain after expansion, the first word is taken to be the name of the command and the remaining words are the arguments.

Redirections are performed as described above (see Redirections).

The text after the = in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.

Command substitution is one of the expansions that take place in (2), and redirection takes place after that in (3). This doesn't have anything to do with subshells, it's purely due to the order in which the shell goes about doing things. (fmt -w 50 < foo.txt) > foo.txt would have resulted in a blank file too.

Do you mean $(fmt -w 50 < foo.txt ) > foo.txt in your example? — shredalert, Jun 24 '19 at 16:40
No, I meant (fmt -w 50 < foo.txt) > foo.txt, where fmt -w 50 < foo.txt is run in a subshell. — muru, Jun 24 '19 at 16:41
Ah, sorry. I forgot that just plain parentheses execute the list inside in a subshell. — shredalert, Jun 24 '19 at 16:43
Thank you for pointing this out. The section could have been worded a bit clearer as it doesn't use the word "precedence" or that the list given is a "sequence". — shredalert, Jun 24 '19 at 16:43

agc · Answer 2 · 2019-06-27T14:30:08.777

2

One could use the sponge util:

fmt -w 50 < foo.txt | sponge foo.txt

A note on the original:

echo "$(fmt -w 50 < foo.txt)" > foo.txt

First executed is fmt, its output gets put in a string, i.e. what's in the double quotes. Nothing else happens until fmt finishes.
echo prints the string to STDOUT.
Meanwhile echo's output is redirected with > to foo.txt.

Steps #2 and #3 occur more-or-less simultaneously.

The original code does have an issue. Command lines have length limits. If the input foo.txt is very large, specifically larger than the length of getconf ARG_MAX, the echo command will either fail or give incomplete results.

edited Jun 27 '19 at 14:30

answered Jun 27 '19 at 13:05

agc

7,223

2

@JeffSchaller, It's a useful partial answer. Meanwhile muru's answer already explained it. More generally, knowing how to do something often helps reinforce understanding the why of it. – agc Jun 27 '19 at 14:05
Thanks for bringing the length limit to my attention. I will keep it in mind. – shredalert Jun 28 '19 at 14:28

Precedence of subshells in relation to redirection?

2 Answers2