7

I've come upon this sed command and I cannot figure out what it is doing. I understand that it is changing a file in place using -i, that it is using a script -e and that the script is $a\, but what is this script doing?

sed -i -e '$a\' filename
  • Looks like it's doing the same thing as GNU awk '1' file - accept an input file with a terminating newline (valid POSIX text file) or without a terminating newline (invalid POSIX text file) and produce an output file that does have a terminating newline (i.e. is a POSIX text file) whether it was present in the input or not. – Ed Morton Dec 14 '21 at 16:23

2 Answers2

14

As others have said, e.g. in How to add a newline to the end of a file?, with GNU sed (and some other implementations), $a\ adds a newline to the end of a file if it doesn’t have one.

Why it does this isn’t so clear, and the documentation doesn’t explain it. However, examining the source code does...

Let’s start with the documentation. $a\ is a variant of a, exploiting a special case of a GNU extension:

As a GNU extension, the a command and text can be separated into two -e parameters, enabling easier scripting:

$ seq 3 | sed -e '2a\' -e hello
1
2
hello
3

$ sed -e '2a' -e "$VAR"

The way a is implemented in sed is with a queue of text to append, tracked in an append_queue. When the time comes to process this queue, in a function called dump_append_queue, the first step is

output_missing_newline (&output_file);

which adds a missing newline if necessary — to ensure that the appended text will be added to separate lines, not to the end of the current line. Then the contents of the queue are appended.

With sed -i '$a\', the missing newline is added if necessary, and then the queue is processed — but the queue is empty, so no additional change is made.

Stephen Kitt
  • 434,908
5

That adds a newline to the last line, if it is absent:

Create a file with no trailing newline

$ printf 'line1\nline2' > file
$ od -c file
0000000   l   i   n   e   1  \n   l   i   n   e   2
0000013

Let's see what that sed does:

$ sed -i '$a\' file
$ od -c file
0000000   l   i   n   e   1  \n   l   i   n   e   2  \n
0000014

What happens if we run it again? Does it add another newline?

$ sed -i '$a\' file
$ od -c file
0000000   l   i   n   e   1  \n   l   i   n   e   2  \n
0000014

Nope.

terdon
  • 242,166
glenn jackman
  • 85,964
  • Could you also explain why it works and especially why it doesn't add a second newline if one is present? – terdon Dec 14 '21 at 16:07
  • Not really. The a command is documented at https://www.gnu.org/software/sed/manual/html_node/Other-Commands.html, but it does not particularly illuminate this edge case. – glenn jackman Dec 14 '21 at 16:11
  • 1
    Strictly speaking, that's probably undefined in the same way as other similar ones are: the POSIX text says for sed that "The input files shall be text files.", and text files are supposed to have the final newline. (also "...many utilities only produce predictable or meaningful output when operating on text files.", which gives a general cop-out.). But it does seem to work with GNU, Busybox and the *BSD sed on my Mac (though that last one adds the newline with about any program.) – ilkkachu Dec 14 '21 at 16:26
  • Some implementations add a newline to the last line, if it is absent (I need to stress this out since not all systems are GNU or Macs). For example, busybox sed '$a\' file always adds a newline to the file, at least on version 1.34.1 from 2021-11-19. – Quasímodo Dec 14 '21 at 17:21
  • @Quasímodo, of course not all systems are GNU or Macs, those are just the ones I had available for testing. If anyone wants to test with something else, the results could be useful. Anyway, like I said, it appears undefined, as anything with sed when given a non-text file as input, so yes, it would have to depend on the implementation. – ilkkachu Dec 14 '21 at 19:50
  • argh, yes, I mistested, Busybox adds the newline even if it exists, so this "trick" doesn't work as generally as I claimed. echo foo | busybox sed -e '$a\' gives foo\n\n. – ilkkachu Dec 14 '21 at 19:56
  • @ilkkachu Sorry for not being clear, my comment was not targeted at yours, but rather at the first paragraph of this answer, that does not determine its scope, and thus could be mistakenly taken by readers as applying to any Sed in general. I mentioned Macos and GNU because those are also the only ones mentioned in the accepted answer to the linked question. – Quasímodo Dec 15 '21 at 11:21
  • @Quasímodo, ah ok :) I wasn't sure since it happened to be the same two I mentioned and read it wrong, sorry. But you prompted me to notice what busybox really does, so thanks for that. – ilkkachu Dec 15 '21 at 13:13