0

AFAIK, cat is an external command and it forks off a new process when executed, just like sh -c or executing a script. With that said I expect cat to use its command environment as it is used by other external command, like 

f=test.txt sh -c 'cat "$f"'

Shouldn't this display the content of the file 

f=test.txt cat $f

?

Note: I am not asking what is someVariable=someValue command. I am asking why first example uses its command variable but not the second one. The way variable expansion is happening in second example it should happen in first example.

haccks
  • 203
  • 4
  • 10
  • 1
    I'm not having any problems with this command. I get the contents of the file "test.txt", as expected. – igal Mar 12 '18 at 21:43
  • 1
    I don't. echo foo > test.txt; f=test.txt cat $f waits for input from stdin rather showing foo. – Philip Kendall Mar 12 '18 at 21:46
  • @PhilipKendall; That's exactly what happening with me. – haccks Mar 12 '18 at 21:47
  • Why it is dupe? I am not asking what is someVariable=someValue command. – haccks Mar 12 '18 at 22:08
  • I misunderstood which statement you were asking about. Your second command won't work because the assignment is applied after the statement is executed, not before. See my solution for a more detailed explanation. – igal Mar 12 '18 at 22:32
  • Your question is why f=test.txt cat $f doesn't display the content of the file (these are your own words) so clearly a dupe. And you don't know the difference between single quotes and double quotes (that would make it also a duplicate of another question so double-dupe) – don_crissti Mar 12 '18 at 22:39

3 Answers3

6

Summary

The command f=test.txt sh -c 'cat "$f"' produces output because the variable assignment f=test.txt occurs before the expansion of the (single-quoted) command-argument 'cat "$f"'. The single-quotes prevent the expansion from taking place until the subcommand cat "$f" is executed.

The command f=test.txt cat $f does not produce output because the variable assignment f=test.txt occurs after the expansion of the (unquoted) command-argument $f.

Why f=test.txt cat $f does not produce any output

First I'll attempt to explain why the command f=test.txt cat $f is not producing any output despite your expectation that it would. Here there might be some slight confusion about the order of evaluation in what's called a simple command.

You may have assumed that the variable assignment in the prologue of the command (i.e. the f=test.txt assignment) occurs before the variable expansion in the body of the command (i.e. the expansion of $f in cat $f). But this is actually not the case. To verify this we can refer to the page on simple command expansion in the Bash Manual or to the subsection on simple commands in the Posix Specification. Both of these references include the following passage:

A "simple command" is a sequence of optional variable assignments and redirections, in any sequence, optionally followed by words and redirections, terminated by a control operator.

When a given simple command is required to be executed (that is, when any conditional construct such as an AND-OR list or a case statement has not bypassed the simple command), the following expansions, assignments, and redirections shall all be performed from the beginning of the command text to the end:

  1. The words that are recognized as variable assignments or redirections according to Shell Grammar Rules are saved for processing in steps 3 and 4.

  2. The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.

  3. Redirections shall be performed as described in Redirection.

  4. Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.

Notice that step 2 is where the variable expansion in the command occurs, but step 1 tells us that the variable assignments are saved until steps 3 and 4. It follows that the expression cat $f is expanded to cat (with no arguments) before the assignment f=test.txt takes place. This explains why you're not getting any output.

For further discussion on this topic see the following posts:

Why f=test.txt sh -c 'cat "$f"' does produce output

Next I'll attempt to explain why the command f=test.txt sh -c 'cat "$f"' does produce output. For this we'll want to look at the full list of general operations performed by the shell:

  1. The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2008. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.

  2. The shell breaks the input into tokens: words and operators; see Token Recognition.

  3. The shell parses the input into simple commands (see Simple Commands) and compound commands (see Compound Commands).

  4. The shell performs various expansions (separately) on different parts of each command, resulting in a list of pathnames and fields to be treated as a command and arguments; see wordexp.

  5. The shell performs redirection (see Redirection) and removes redirection operators and their operands from the parameter list.

  6. The shell executes a function (see Function Definition Command), built-in (see Special Built-In Utilities), executable file, or script, giving the names of the arguments as positional parameters numbered 1 to n, and the name of the command (or in the case of a function within a script, the name of the script) as the positional parameter numbered 0 (see Command Search and Execution).

  7. The shell optionally waits for the command to complete and collects the exit status (see Exit Status for Commands).

So you can see here that calling a function/built-in/executable/script (step 6 on this list) occurs after parsing of the simple command. Therefore the assignment f=test.txt occurs before the program execution sh -c 'cat "$f"'. And because the argument is single-quoted it is only parsed after the command executes. Therefore the subcommand expands to cat "test.txt".

igal
  • 9,886
  • With the rules above f=test.txt cat '$f' should work? – haccks Mar 13 '18 at 07:42
  • @haccks No, because in the command f=test.txt cat '$f' the variable f will never be expanded. That command is going to try to apply cat to a file literally named $f, i.e. the same as if you'd escaped the dollar-sign cat \$f. The reason that the command f=test.txt sh -c 'cat "$f"' works is because of the sh command. That causes the argument cat "$f" to undergo its own round of parsing. – igal Mar 13 '18 at 12:37
  • I read here, it says that expansion will happen in case of TEST=foo echo $TEST. Then why not in the above example of mine? Is there any rule where variables will be expanded and where it will not? – haccks Mar 13 '18 at 13:40
  • @haccks You should say specifically which command you're talking about, otherwise I can't be sure what you mean. The command f=test.txt cat '$f' has the variable expression contained in single-quotes. Single-quotes prevent variable expansion from taking place. In the command TEST=foo echo $TEST the variable expansion occurs because the variable expression is unquoted; however the expansion occurs before the assignment, so it likely does not have the intended effect. To get the intended effect you could do this TEST=foo sh -c 'echo $TEST'. – igal Mar 13 '18 at 14:16
  • That means in case of sh -c there are two round of parsing apart from the parsing of command TEST=foo sh -c 'echo $TEST' which will be done by parent shell , is it? – haccks Mar 13 '18 at 14:20
  • 1
    @haccks That's correct. When the command sh -c 'echo $TEST' is invoked it executes the command echo $TEST inside of a subshell. This kicks off another round of parsing and expansion. The variable assignment TEST=foo has already taken place and is automatically exported to the subshell environment, as described in the subsection on simple commands. – igal Mar 13 '18 at 14:29
  • AFAIK, sh -c doesn't invoke subshell, but a child process. If sh -c will invoke a subshell and parsing is happening twice then that will hold true for (TEST=foo echo '$TEST') (which seems that parsing only once). – haccks Mar 13 '18 at 14:33
  • @haccks I believe that sh -c does invoke the command in a subshell. A subshell also happens to be a child process - this isn't a contradiction. The parenthesized command (TEST=foo echo '$TEST') (i.e. a compound command) also spawns a subshell. – igal Mar 13 '18 at 15:32
  • @haccks Anyway, I don't think your reasoning is correct. The expression (TEST=foo echo '$TEST') would be equivalent to sh -c "TEST=foo echo '\$TEST'", but it is not equivalent to the expression in question, namely TEST=foo sh -c 'echo $TEST'. In the parenthetical syntax (i.e. with a compound command) this might have looked like TEST=foo (echo $TEST), but the shell parser doesn't support this syntax for compound commands. – igal Mar 13 '18 at 15:32
  • This says that sh -c does not spawn a subshell but it's just a child process (last example). Also see this answer. – haccks Mar 13 '18 at 15:45
  • @haccks It's true that I wasn't wasn't distinguishing between a child shell and a subshell, but I really don't think that's relevant here. You used the term "child process" before, which is what I was responding to. In any event, the examples that we've been discussing involved exporting a variable assignment, so the sub-shell vs child-shell distinction shouldn't affect anything in a critical way. I believe that all of my previous comments explain the situation. If you think the child-shell vs subshell distinction is important feel free to elaborate on that. – igal Mar 13 '18 at 16:21
  • @igal: Why do you refer to sh -c 'cat "$f"' as a “function execution”? Step 6 says “The shell executes a function …, built-in …, executable file, or script”, and sh and cat are clearly executable files and not functions. – G-Man Says 'Reinstate Monica' Mar 14 '18 at 20:25
  • @haccks: Since nobody else seems to have pointed this out yet (and, if somebody has and I overlooked it, I apologize) — the problem with your quote from peterph’s answer to “How do I set an environment variable on the command line and have it appear in commands?” is that you’re misquoting it. That answer says that TEST=foo; echo $TEST will output foo. But now you’re asking about TEST=foo echo $TEST — *not the same command.* … (Cont’d) – G-Man Says 'Reinstate Monica' Mar 14 '18 at 20:26
  • (Cont’d) …  While you’re still learning shell grammar, you might want to get into the habit of using copy and paste, because tiny differences can be more significant than you might guess. Just as '… $var …' is different from "… $var …", so also is assignment ; command different from assignment  command. – G-Man Says 'Reinstate Monica' Mar 14 '18 at 20:27
  • @G-Man I was using "function" generically - not a good idea in this case, given the context. I've updated my post to be more precise. – igal Mar 14 '18 at 20:44
  • @G-Man; Man! You are hunting me down all over the SE sites!!! :D I appreciate your effort and willingness to help. Actually I wasn't misquoted TEST=foo; echo $TEST by TEST=foo  echo $TEST, it was intentional. This example is given by OP in the question. – haccks Mar 14 '18 at 20:54
2

Variable assignments on a shell command affect only the environment passed to that command, not the executing shell's environment.

You can see this more directly with 

$ f=1
$ f=2 echo $f
1
$

The echo ran in an environment where f was set to 2, but the assignments ran in the (under-construction) command's environment, not the shell environment used to construct the command.

It's only assignments without a command that affect the running shell's environment, and it's the shell's environment that's used to construct commands.

jthill
  • 2,710
0

Because in your second example, $f is expanded by your shell, not by cat. Assuming f isn't set otherwise in your shell's environment, $f expands to the empty string and cat then reads from stdin as it would if given no arguments.

In your first example, the $f is inside single quotes so it is not expanded by original shell, but is expanded by the instance of sh which sees cat "$f" and has f set in its environment.

Philip Kendall
  • 593
  • Why variable expansion is not happening in first example by current shell? AFAIK, shell expands variables before it runs the command. – haccks Mar 12 '18 at 21:51
  • 1
    @haccks the shell expands variables before it even sets f, when you run f=test.txt cat $f; so $f expands to the empty string in your case (or whatever it was already set to), then f is assigned the value test.txt, then cat is run. See the suggested duplicate for details. – Stephen Kitt Mar 12 '18 at 22:01
  • @StephenKitt; I am asking why it is not happening in the first case f=test.txt sh -c 'cat "$f"'? – haccks Mar 12 '18 at 22:07
  • Are you saying f=test.txt sh -c 'cat "$f"' doesn't display the contents of test.txt? It certainly does on any machine I have trivial access to. – Philip Kendall Mar 12 '18 at 22:10
  • @PhilipKendall; No. I am not saying that. What I am saying is "why variable expansion happening in f=test.txt cat $f while not in f=test.txt sh -c 'cat "$f"'? – haccks Mar 12 '18 at 22:12
  • OK, I've added something to the second section of my answer. – Philip Kendall Mar 12 '18 at 22:22