0

This question is about rendering templates with sed.

I have the template file nginx_app:

server {
    root ${drt}/${domain}/;
    server_name ${domain} www.${domain};
}

I use that file in another script:

#!/bin/bash
domain="$1" && test -z ${domain} && return
sed "s/\${domain}/${1}/g" "nginx_app" > "/etc/nginx/sites-available/${domain}.conf"

The purpose is to have a example.com.conf as a rendered template, based on nginx_app, of course.

Consider this code from above:

sed "s/\${domain}/${1}/g" "nginx_app" > "/etc/nginx/sites-available/${domain}.conf"

I understand this code does the following:

  1. It changes the (yet unexpanded) string ${domain} per the script argument of ${1} (example.com), and that's done somewhere in RAM, but without changing the nginx_app file itself.
  2. It renders the nginx_app template by redirecting its rendered version into /etc/nginx/sites-available/${domain}.conf.

Assuming I was accurate here, I must say that these actions were very non intuitive for me from the code. Was I accurate in my description of that code?

Please correct my description if it wasn't enough accurate.

Arcticooling
  • 1
  • 1
    Without the -i flag, the results of any sed command will be written to standard output. In this instance, you are redirecting standard output to another file. – Raman Sailopal Feb 06 '18 at 13:33
  • sed basically reads its input line by line and performs any transformations on the lines before writing the line out, in this case to standard output. So in this case it at most needs to store one line at a time in memory. Any computer program does its work in memory, that shouldn't come as a surprise. – wurtel Feb 06 '18 at 13:37
  • I meant to RAM, not memory in general, I edited. – Arcticooling Feb 06 '18 at 13:59
  • @RamanSailopal sounds to me like this should be an answer. You explained it very clearly to me. – Arcticooling Feb 06 '18 at 13:59

3 Answers3

1

Yes, your understanding in the second part of the text is correct.

sed, like many other Unix tools, can be said to act like a filter. A filter takes input, modifies it, and produces output. In many cases, the utility that is doing the filtering may not even be aware that it's reading from, or writing to, a file. In fact, it may read from, or write to, another filter utility, directly. This is transparent to the utility itself and something that the shell is responsible for setting up.

Ignoring the fact that the following is using cat unnecessarily,

cat <file | sed 's/^\([^=]*\)=\(.*\)$/#\1=\2 # old/' | tr -s ' ' >newfile

None of the utilities above are aware of reading from a file, or writing to a file. cat reads from its standard input stream (which the shell arranges to contain the data from file), probably a line at a time, and writes its output to its standard output stream (without modifications).

sed reads from its standard input stream, which the shell has already connected to the output stream from cat, does some modifications on each line of input, and writes the result to its standard output stream (which is connected to tr).

tr just writes to standard output as well, but the shell sets up the redirection in such a way that the output goes into the file newfile.

In each step of this process, each utility is likely to only hold what's minimally necessary of the data that passes through the pipeline (in buffers allocated in RAM). This means that results may start being written to newfile (in this example) before the data in the file file has even been completely read.

Your specific example:

sed "s/\${domain}/${1}/g" "nginx_app" > "/etc/nginx/sites-available/${domain}.conf"

What's happening:

  1. The shell parses the command line and finds the variables and the redirections.

  2. The file /etc/nginx/sites-available/${domain}.conf (with $domain expanded to the value of that variable) is either truncated (or emptied) if it exists, or created as an empty file if it doesn't exist.

  3. The sed utility is invoked with the operands s/\${domain}/${1}/g (where $1 has already been expanded to its value, but ${domain} hasn't, since the $ is escaped with \) and nginx_app. The shell additionally connects the standard output of sed to the file that you are redirecting into.

  4. Since sed got two operands, it will assume that the second one is a file to read from. It opens it and reads it line by line while applying the editing commands in the first operand.

  5. Any output goes into the file that you redirect to.

Since people are mentioning sed -i:

The -i flag to sed makes sed do perform its changes on the given file in place, that is, it will not write to its standard output but the result of its changes will be reflected in the same file that was used for input.

Internally, sed does this by writing to a temporary file which it later replaces the original file with.

The -i flag takes an (optional for GNU sed and a few other implementations of the utility, mandatory on others) argument:

sed -i .bak '...some sed script...' filename

This causes the original file to be backed up with the given string as its new filename suffix.

I generally try to discourage people from using -i with sed, for a few reasons:

Instead, I recommend that one does something on the lines of

sed '...some sed script...' file >file-new && mv file-new file

... and only add the && mv part when the sed part has been properly tested.

The && mv means "if the previous command terminated successfully (without error), rename this file".

Kusalananda
  • 333,661
  • Thanks! Did you mean in The parses the command line to the shell interpreter parses the command? – Arcticooling Feb 06 '18 at 14:22
  • 1
    @user9303970 Yes! I fixed it just as you commented. – Kusalananda Feb 06 '18 at 14:22
  • I think it's worth adding a sentence about the fact that without the -i flag, results of sed commands will be writen to stdout. It helped me very much to understand even your answer, Kusalananda, after I read that in Raman's answer... – Arcticooling Feb 06 '18 at 15:45
  • @user9303970 I did not mention that detail since it was not mentioned in the question. Additionally, the -i flag is not a standard flag (it works subtly different on different Unix systems). But I'll add a short note about it... shortly. – Kusalananda Feb 06 '18 at 15:59
  • @Philippos I appreciate the edit, but using sed y in place of tr is not what I would do there as this is exactly the kind of transformation tr is all about. I would use the sed y command inside a larger sed script, but I really see no point in using it on its own. – Kusalananda Feb 07 '18 at 06:54
  • That's what I tried to say: You shouldn't pipe if you can integrate everything into one script. I don't want to think readers why should I do it this way. Maybe you change your example to something where piping has a reason: Instead of cat | sed | tr you do something sort | sed | shuf, which doesn't need an excuse for being bad style. – Philippos Feb 07 '18 at 07:21
  • @Philippos Ah, I see what you're saying now. I really just used random filters in the pipeline without much thought as to what they did. I will fix that by making the pipeline do something useful (just need to think first). – Kusalananda Feb 07 '18 at 07:35
1

Without the -i flag, the results of any sed command will be written to standard output.

In this instance, you are redirecting standard output to another file (/etc/nginx/sites-available/${domain}.conf).

Arcticooling
  • 1
Raman Sailopal
  • 1,432
0

It sounds like you're asking why you have to redirect the output into a file rather than sed writing into that file directly. If so, there are a few reasons for this.

  1. You actually can do this with the -i argument to sed.
  2. This is not the default behavior as it doesn't follow the common unix philosophy of designing programs which talk to other programs (https://en.wikipedia.org/wiki/Unix_philosophy).
    The idea behind many unix shell utilities is that you chain them together (foo | bar | baz). This allows you to perform complex processing on the data without having to buffer it to disk between each step.
phemmer
  • 71,831