Understanding the "process substitution" concept

Question

I seek a simple explanation about the term "process substitution" aimed for general audience / non professional sysadmins, which do server environment maintenance for personal projects once in a while and not daily for customers or companies, if its okay to ask for it here.

• What problem/s this concept was invented to solve
• What process is being substituted and with what (the name might mislead)?
• Is it also named "preserving stdin" and if so why? If the stdin device holds standard input which can be piped to another command, what is there to "preserve" further than that?

This might help me to understand the following command:

bash <(wget -O - URL)

_{Note for newcomers: -O - tells wget to write data to stdout;}

Answer 1

Important fact: there are two shells involved

First note there are two shells involved. There is the outer Bash you work in, you invoke (type) bash <(wget -O - URL) in it. And there is the inner Bash that runs as a child process of the outer Bash when the command gets executed.

In other words the outer Bash accepts bash <(wget -O - URL) as a command and it spawns the inner Bash. I will distinguish the two where appropriate.

---

Important fact: there is more than one way to run code with Bash

There are few ways to make (an inner) Bash run some code:

1. bash can read code from its stdin:

   echo 'date; sleep 2; date' | bash
   

   The stdin can be the console. E.g. an interactive bash you normally work in reads from the console but it actually reads from its stdin because the console is its stdin.

2. bash can read code from a file:

   bash /path/to/some/file
   

   (The path may be relative.)

3. bash can read code from a command-line argument:

   bash -c 'date; sleep 2; date'
   

---

The context

In a comment you said the context is this question. My answer there advised to replace:

wget -O - URL | bash

with

bash <(wget -O - URL)

The first command makes the inner bash read from its stdin. What you seem not to have realized is that the second command makes the inner bash read from a file that is not its stdin.

---

What happens to <(…)

When you run bash <(wget -O - URL) in Bash, the outer Bash replaces <(…) with the path to some file. After the replacement the actual command being run may be like:

bash /dev/fd/63

This spawns the inner Bash. It opens /dev/fd/63 like it would open /path/to/some/file.

Thanks to the "magic" of process substitution /dev/fd/63 is a pipe already connected to wget -O - URL.

^{Side note: on a system where /dev/fd is unavailable the outer Bash would use a truly named pipe (e.g. /tmp/sh-np.pldKay) to set this up.}

---

The similarity

The connection from wget to (the inner) bash is similar in the two cases. When you run wget -O - URL | bash, bash reads code from its stdin which is a pipe wget writes to. When you run bash <(wget -O - URL), bash reads code from another file (i.e. not from its stdin) which is a pipe wget writes to. In both cases the outer Bash sets the pipe up.

---

The difference

The difference is with process substitution the stdin of the inner Bash is not used to pass code to it. The stdin can be used for another purpose. E.g. the stdin can be the console and read in the code can read from it.

---

Answers to your explicit questions

• What problem/s this concept was invented to solve

  When a command expects a pathname (path to a file) and you want to pipe something in instead of providing a regular file, you can create a named pipe and achieve the goal. This requires actually creating a pipe (mkfifo), piping something to it (in the background or in another console), running the command and finally removing the named pipe (rm).

  With process substitution Bash handles the piping. It's convenient.

• What process is being substituted and with what?

  In bash <(wget -O - URL) the <(wget -O - URL) syntax is substituted with a path to some file, only then the command is executed (and it's bash /dev/fd/63 or similar). The outer Bash prepares the file, so reading from it means reading what wget -O - URL (the process) writes. The file is actually a pipe, not a regular file.

• Is it also named "preserving stdin" and if so why?

  "Preserving stdin" happens, but it's not an alternative name for process substitution. It's not a formal name for what happens either.

  When you run (inner) bash … in (outer) Bash, the inner Bash gets its stdin inherited from the outer Bash (i.e. it's the same file). The stdin may be the console.

  But if you run wget … | bash … instead, the inner Bash will see data from wget on its own stdin. Now some part of the inner Bash (e.g. the read builtin) or some child of the inner Bash may want to read something from stdin. They expect input from a console or whatever, but not the code the inner Bash should execute. But because the stdin of the inner Bash is a pipe from wget, builtins will use it and child processes will inherit it as their stdins. They won't inherit the stdin of the outer Bash.

  By using process substitution instead of piping code via the stdin of (the inner) bash, you make the stdin of the outer Bash available for builtins and children of the inner Bash. You preserve the stdin of the outer Bash for builtins and children of the inner Bash. At the same time you protect the stream of code the inner Bash reads and executes from being read by its builtins or children.

  This works only because bash allows you to provide code in a file specified as a command line argument. If (the inner) Bash supported reading code from stdin only, then you would have to provide code via its stdin. In such case process substitution could not solve the issue you had.