When printing a variable that contains newlines, why is the last newline stripped?

Question

Contents of file.txt (no weirdness, text file as defined by POSIX)

iguana
gecko
anole

Sample script:

#!/bin/sh

string="$(cat file.txt)"

printf '%s' "$string"

Sample output:

[coolguy@somemachine ~]$ ./script.sh
iguana
gecko
anole[coolguy@somemachine ~]$

What happened to the last newline? Why are all are newlines except the last one preserved? It seems like we shouldn't have to use echo to add a newline if there should already be one there.

@Jesse_b - according to POSIX "a text file, contains characters organized into one or more lines" and "a line is a sequence of zero or more non-newline characters plus a terminating newline character." — don_crissti, May 30 '18 at 21:08
Effectively dupe https://unix.stackexchange.com/questions/17747/why-does-shell-command-substitution-gobble-up-a-trailing-newline-char and https://unix.stackexchange.com/questions/17732/where-has-the-trailing-newline-char-gone-from-my-command-substitution — dave_thompson_085, May 30 '18 at 22:58
Might be worth mentioning that, for the same design rationale, here-string redirections add a newline at the end of the string provided, like in cat <<< foo. — JoL, May 31 '18 at 04:46

score 25 · Accepted Answer · edited May 30 '18 at 21:53

It's not the printing, it's the command substitution that does that. It's defined to do that. From the POSIX description:

The shell shall expand the command substitution by executing command in a subshell environment and replacing the command substitution with the standard output of the command, removing sequences of one or more {newline} characters at the end of the substitution.

Note that it removes all trailing newlines, not just one.

In a somewhat common case, you'd use command substitution to capture a one-line output, say osrev=$(uname -r). The utilities usually print a trailing newline, for user convenience on the command line. But in a shell script, you might want to use that string as part of another one, say a filename: filename=blahblah-$osrev.dat. And in that case, the trailing newline would only be a nuisance.

And of course, a plain echo will add a final newline in any case.

If you want the contents of the file as-is in the variable, then the common workaround is to add an extra character in the command substitution, and remove that later:

printf "foo\nbar\n\n" > file
string=$(cat file; echo x)
string=${string%x}
printf '%q\n' "$string"

that outputs $'foo\nbar\n\n', showing both trailing newlines present.

Depending on what you intend to do with the data, there may be other ways. E.g. a while read loop, or Bash's mapfile, if you happen to want to process the file line-by-line.

This question has been marked as a duplicate. Is it good practice to delete a question after it has been marked as such? — Harold Fischer, Jun 18 '18 at 20:16
@HaroldFischer, there's a whole bunch of questions marked duplicates on the site, so no, not really. And I don't think you even could delete it, since there's an upvoted answer. See deleted-questions. — ilkkachu, Jun 18 '18 at 20:45

When printing a variable that contains newlines, why is the last newline stripped?

1 Answers1