How does exec > work in AWS EC2 user data?

Question

I was writing a Terraform module for AWS EC2 that involved executing a bash in the user data section. While I was developing I had an issue in the script I wrote but neither AWS nor Terraform provided any logs for the errors in I got until I found this line in AWS support forum:

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

This line writes all the output of the script executed in user_data to /var/log/user-data.log successfully but I don't understand the whole line. I know exec > writes all the output to a file, which in this case >(..) but I don't understand why it's using tee or the need for a pipeline there.

score 7 · Accepted Answer · answered Sep 28 '21 at 10:16

Let's break that in to pieces:

exec applies redirections on the shell itself, so indeed exec > somefile would direct all later output from the shell and its children to that file.

Here, instead of a simple file name, we have the process substitution >( command... ). It tells the shell to run the command inside, and to create a some filename that's connected to the input of that command. foo > >(bar) is a bit like foo | bar, except that process substitutions can be used for more complex cases too, and don't invoke subshells like the pipeline does. (<(command...) would be the reverse, it would make the output of the command appear as if from a file.)

The command inside is tee filename | logger .... tee copies its input both to stdout, and to the given file(s), and logger sends its input to the system log. (The redirection 2>/dev/console would apply to error output from logger, probably it would print some errors if the system log doesn't work.)

Then there's the 2>&1, which tells the shell to redirect error messages (stderr) also to that same process substitution.

So, after the redirection, everything the shell outputs (normal output and error output) goes (via tee) both to the file and (via logger) to syslog.

(I don't think any of that is specific to AWS or Linux, unless of course there's some particularities about the use of logger on AWS.)

The exec combined with process substitution part is specific to bash - AFAIK, only bash allows that with exec. Other Bourne-like shells require you to set up and use a fifo if you want to do that. See my answer to How can I redirect all output of a script to a file and replace passwords? — cas, Sep 28 '21 at 14:28
@cas Zsh allows it as well. Ksh93 doesn't. Mksh doesn't implement process substitution. So it's 2/3 for shells with process substitution. — Gilles 'SO- stop being evil', Sep 28 '21 at 15:02

How does exec > work in AWS EC2 user data?

1 Answers1