30

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens

  x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p 
# Output is: 610a62 
# The trailing newline from the 'echo' command
#   has been "deleted" by Command Substitution

I assume there must be some very significant reason for a shell action, namely Command Substitution, to actually delete some data from the command output it is substituting...
but I can't get my head around this one, as it seems to be the antithesis of what it is supposed to do.. ie. to pass the output of a command back into the script process... Holding back one character seems weird to me, but I suppose there is a sensible reason for it... I'm keen to find out what that reason is...

Community
  • 1
Peter.O
  • 32,916

5 Answers5

22

Because the shell was not originally intended to be a full programming language.

It is quite difficult to remove a trailing \n from some command output. However, for display purposes, almost all commands end their output with \n, so… there has to be a simple way to remove it when you want to use it in another command. Automatic removal with the $() construction was the chosen solution.

So, maybe you'll accept this question as an answer:

Can you find a simple way to remove the trailing \n if this was not done automatically in the following command?

> echo The current date is "$(date)", have a good day!

Note that quoting is required to prevent smashing of double spaces that may appear in formatted dates.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Stéphane Gimenez
  • 28,907
  • 2
    If what you say is true, then I would be absolutely stunned. If there is a shopt to change your described default behaviour, then I'd be happy again... I find it difficult to think that Bash/Linux/Unix would pollute such a critical function as "capturing stdout" just because date doesn't have a with/without '\n' option... The obvious fix would be that bash (or other shell) has a shopt for this... Do you know of any? .. and do you have any reference links that speak of the whys and wherefores of this isssue? ... and why doesn't date have a \n option.. It should! – Peter.O Jul 31 '11 at 16:02
  • 3
    Command substitutions appeared in the first version of the Bourne shell in the "7th edition" of Unix in 1979. It was already a big revolution and I guess (I was not born) that nobody cared about keeping trailing '\n' or not. Yes, you can read the manpage in less than 10min, and it seems nothing has changed about command substitution until then. Except for the preferred $() alternative syntax. – Stéphane Gimenez Jul 31 '11 at 17:01
  • 1
    Thanks.. I thought this may be a legacy issue, and it certainly is shaping up to the fact that I'd better start watching my Command Substitutions more carefully. Until today I must have been surviving on a wing an a prayer.. Some of those "mysterious happenings" I've encountered are starting to make sense now :) ... So in a reverse case of your "date" posit, if I want to keep my trailing \n, I must code something like this :( { echo -n "The current date is "; var="$(date; echo -e x)"; var="${var%x}"; echo -n "$var"; echo ", have a good day!"; } .. so be it :) – Peter.O Jul 31 '11 at 18:08
  • PS re my above comment: Of course, just date would work, but I just used date as an example of any command.. – Peter.O Jul 31 '11 at 18:39
  • Your 'question' about date was the strongest promt towards my understanding of 'why' Command Substution does what it does, so this has been the 'best' answer to my question... and I certainly also needed the other good answers as well.. – Peter.O Jul 31 '11 at 22:41
  • https://stackoverflow.com/questions/613572/capturing-multiple-line-output-into-a-bash-variable – sancho.s ReinstateMonicaCellio Sep 14 '18 at 12:39
  • "Because the shell was not originally intended to be a full programming language." [citation needed] –  Oct 26 '20 at 18:57
20

It's part of the standard:

The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.

Of course, the standard is probably written this way because that's how ksh did it or something, but it is the standard, and it is documented behavior. If you don't like it, use Perl or something else that will let you keep the trailing newlines.

Chris Johnsen
  • 20,101
Steven Pritchard
  • 2,028
6

Well, it makes sense to me. The newline is only there in the first place, in normal command output, so that the prompt appears after the command completes on a new line. The newline isn't part of the original output in most cases, it's there to tidy the screen up. When you're parsing output from a command the newline at the end is usually troublesome. Why does the wc command output 2 lines of text? Oh, it doesn't, it outputs one followed by a newline. When you parse wc you don't want to be worrying about the fact that there's 2 lines of output - there aren't really, there's only one.

EightBitTony
  • 21,373
  • @EightBitTony: My question is not in any way related to displaying something in the terminal.. That is quite straight forward. If there is a newline to display, then it will display the newline.. The point in question here, is that a \n in the original stdout stream is removed by Command Substitution. ie. my original data ( very important data ) has trailing newlines removed, even when the data is wrapped in "quotes". I reasonably understand how and why Word Splitting works, but I have absolutely no idea why Command Substitution strips only newlines and only trailing ones. – Peter.O Jul 31 '11 at 15:18
  • 1
    Nothing you have said changes my view. More often than not, the final newline in any output is there purely for display purposes, not as part of the data. – EightBitTony Jul 31 '11 at 15:46
  • I agree with your view, and have all along... Yes the newline at the end of output is for convenience... but my issue has nothing to do with that... I am asking: Why does Command Substitution forcibly remove it? ... These are two different issues.. Maybe my question should be: Is there a built in workaround? . because having to code for this issue by adding a trailing dummy char, sucks! ... I'll do it if I have to, but this is the first time I've been stymied by bash (and I'm in a state of shock :).. It does so much so well... – Peter.O Jul 31 '11 at 16:13
  • As mentioned in another answer, it's part of the standard. For me, I think it does it that way because when you post-process data you only want to see the data, not the convenient newline. It's because the shell expects you to be handling the data in a pipe, and the newline is not data. Any more and we need to take this to a discussion. – EightBitTony Jul 31 '11 at 17:19
  • Thanks.. I've pretty well got the idea now.. It seems to be a case of Command Substitution simply leans towards dropping the trailing newlines, rather than maintaing them.. There must be some wisdom in that move which I havent "felt" yet, but I used to think that the overwhelming popular use of all lo-case letters in filenames was a bit strange/unnecessary, but I was thinking just the other day that it is actually quite a comfortable system.. I'll probably see (more deeply) the rhyme and reason of Command Substitution's behaviour, some day... – Peter.O Jul 31 '11 at 18:28
1

I was running into same issue, and found below example. Seems quoting helped the situation, at least it did for me:

http://tldp.org/LDP/abs/html/commandsub.html.

dir_listing=`ls -l`
echo $dir_listing     # unquoted

# Expecting a nicely ordered directory listing.

# However, what you get is:
# total 3 -rw-rw-r-- 1 bozo bozo 30 May 13 17:15 1.txt -rw-rw-r-- 1 bozo
# bozo 51 May 15 20:57 t2.sh -rwxr-xr-x 1 bozo bozo 217 Mar 5 21:13 wi.sh

# The newlines disappeared.


echo "$dir_listing"   # quoted
# -rw-rw-r--    1 bozo       30 May 13 17:15 1.txt
# -rw-rw-r--    1 bozo       51 May 15 20:57 t2.sh
# -rwxr-xr-x    1 bozo      217 Mar  5 21:13 wi.sh
christian_hercules
  • 19
  • 2
    Here's what's happening here. Let's say you have a variable, $str, containing the string "a b c". If you pass $str to echo without quotes (echo $str), echo will receive a, b, and c as three separate arguments. Your shell doesn't care about the whitespace in between the arguments. echo will then print those arguments with spaces separating each one, so you get "a b c". If you pass the variable to echo with quotes around it (echo "$str"), the shell sees it as one argument and echo receives it as such, so the whitespace is not collapsed. The same applies to newlines. –  Apr 19 '13 at 01:23
  • 1
    The problem that the original poster is having is that the trailing newlines (just the very last ones) of his commands are disappearing. Unless passed the -n flag, echo adds a trailing newline to its output, so both of the echos in your example have trailing newlines. However, if you try echo -n $dir_listing or echo -n "$dir_listing", you will see that no trailing newline is printed with or without quotes around $dir_listing, which suggests that the problem is with the command substitution. –  Apr 19 '13 at 01:31
  • 2
    tldp is a *badly* outdated place to learn about shell scripting. Try the Wooledge Bash Wiki instead. http://mywiki.wooledge.org/BashGuide – Wildcard Oct 06 '17 at 04:42
-1

why does echo "hello " (without the quotes) gobble up the space? the value of IFS characters are seen by the shell as being delimiters, therefor, the trailing ones (that don't delimit anything) are being removed. commend substitution is just plain executing the commends in a sub-shell, so it follows the same logic.

Philomath
  • 2,877
  • @Philomath: Both x="hello  " and x="hello     " will lose all their trailing spaces if the are displayed via echo $x... However only the very last newline of the Command Substituton is lost. – Peter.O Jul 31 '11 at 14:14
  • strange, I don't see any difference in my bash (verifying with xxd) – Philomath Jul 31 '11 at 14:25
  • I just checke on this... It does remove all trailing newlines... I was experimenting with IFS when I got that impression, so let's cancel that one :).. but the question still remains... It is an fundamental part word splitting to condense multiple spaces to a single space, and to totally remove leading and trailing whitespace.. I can understand that, but I certainly don't understand why, with Command Substitution, it only strips \n and not all whitespace (as with word splitting), and why only the trailing end?.. and above all: "Command Substitution" is only partially substituting.. Why? – Peter.O Jul 31 '11 at 14:49
  • 5
    IFS has nothing to do with stripping newlines at the end of command substitutions. – Gilles 'SO- stop being evil' Jul 31 '11 at 21:09