0

Why the shell structure $(...) removes the ending newline character?

#!/bin/bash

S='a
b
'
printf '%i [%s]\n' "${#S}" "$S"


T=$(printf '%s' "$S")
printf '%i [%s]\n' "${#T}" "$T"

That prints this:

4 [a
b
]
3 [a
b]

The variable S contains 2 newline characters. The first printf prints it correctly. Then the second printf prints S and assigns the exact same result to T via the $(...) evaluation. Finally the third printf shows that the last newline character is missing in T! Why?

I have the same result with sh, ksh and Zsh.

Olivier Pirson
  • 147

2 Answers2

2

This is documented behaviour; man bash:

Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.

Of course, you could argue that this does not answer a "Why?" question.

I am not aware of an official explanation. My guess is that it makes sense because it is what you usually need. Adding newline is also easier than removing newline.

# This is how people are used to print text
echo foo

In the other scenario this would end in two newlines at the end


echo foo "$(whatever)"


could behandled with echo -n or printf but who would consider that an improvement?

Another reason is that some commands add a newline at the end and others don't. This behaviour relieves the user of the burden to care about this difference.

avoiding this effect

You can do this in order to get the real command output:

T=$(whatever; echo x)
T_real="${T%x}"
Hauke Laging
  • 90,279
  • Thanks. Is there a way to get the exact result? Of course I can add a newline myself. But I want a newline only when there is really a newline in the content. (I have a workaround. Always add a final character and then remove it. But it is dirty.) – Olivier Pirson Jun 19 '23 at 20:15
  • @OlivierPirson See my edit – Hauke Laging Jun 19 '23 at 20:21
  • Nice. I had more or less the same idea but yours is better. And compatible with sh. – Olivier Pirson Jun 19 '23 at 20:28
  • x is not the best choice of a character to append there. See the linked Q&As for details. – Stéphane Chazelas Jun 20 '23 at 17:00
  • @StéphaneChazelas That is an interesting problem but none of your answers gives a hint how that could become a problem in practice. You add an x, you remove an x. Nothing can go wrong there. Unless you change the encoding between these consecutive operations which would be closer to sabotage than to programming. – Hauke Laging Jun 20 '23 at 23:53
  • It's to piss of Perl programmers who were hoping to demonstrate their chops. – Kaz Jun 21 '23 at 05:39
  • 1
    shell: keep trailing newlines ('\n') in command substitution gives a practical example. Adding a x can complete an unfinished / truncated character like the ū in a locale using the BIG5HKSCS charset (as used in the zh_HK locale on Ubuntu 22.04 at least for instance) as mentioned there. And it's that ū character, not the last byte of it that ${T%x} would remove. Adding a . or / instead doesn't have the problem as their encoding is guaranteed not to be found in other characters. – Stéphane Chazelas Jun 21 '23 at 05:46
2

Why? Unusefully, because the standard says so:

...replacing the command substitution with the standard output of the command, removing sequences of one or more <newline> characters at the end of the substitution.

Realistically, probably because it always worked like that (considering that POSIX mostly codifies existing behaviour, as far as I understand).

More usefully, it might be so that you can do e.g. this:

echo "$(date +"%F %T") $(hostname): something happened..."

instead of

t=$(date +"%F %T")
t=${t%
}
h=$(hostname)
h=${h%
}
echo "$t $h: something happened..."

That is, most commands that print some single value likely print the newline at the end so that they're more friendly to use on the command line. (Without the shell having to detect the cursor not being at the beginning of line after a command; zsh does that, most others don't.) But if you're going to use that value in a script for something else, you probably don't want the newline, just the content. This doesn't only go for printing, but also for passing the value as an argument to some command.

The common workaround is to append an extra character and to remove it:

foo=$(printf 'foo\n'; echo .)
foo=${foo%.}
echo "<$foo>"

Or, if you only have a printf inside the command substitution, use printf -v where available:

printf -v foo 'foo\n'
ilkkachu
  • 138,973