Bash option to make command substitution preserve trailing newlines

Question

I'm hoping that bash comes with an option that stops command substitution from stripping trailing newline characters. Is there? If there isn't, then do any shells exist that are bash-like, and for which standard—POSIX, I guess—sh code would run in, that has such an option or perhaps some special syntax for command substitution that doesn't strip newlines (the latter would be preferable)?

Answer 1

There's no option to do this directly (that I know of), but you can fake it by adding a protective non-newline character in the command substitution, and then removing that afterward:

var="$(somecommand; echo .)"  # Add a "." (and one newline that command
                              # substitution will remove) *after* the
                              # newline(s) we want to protect
var="${var%.}"  # Remove the "." from the end

Note that the exit status of somecommand is lost in the process, since both echo . and var=${var%.} set the status (normally to 0). If you need to preserve it until the end, you need to add a little juggling (w/ credit to Kusalananda):

var="$(somecommand; err=$?; echo .; exit "$err")"  # Make the subshell
                                                   # exit with somecommand's
                                                   # status
commandstatus=$?    # Preserve the exit status of the subshell
var="${var%.}"
# You can now test $commandstatus to see if the command succeeded

Note that the err and commandstatus variables both store the command's exit status, but they aren't redundant, since err only exists in the subshell created by $( ), and commandstatus only exists in the parent shell. You could use the same name for both, but that might add confusion.

BTW, if you don't need to preserve the status until after the "." is trimmed, you can skip the commandstatus part:

if var="$(somecommand; err=$?; echo .; exit "$err")"; then
    # somecommand succeeded; proceed with processing
    var="${var%.}"
    dosomethingwith "$var"
else
    echo "somecommand failed with exit status $?" >&2
fi

^{Beware that depending on the locale, you can't just use any character in place of `.`. The encoding of `.` is guaranteed by POSIX not to be found in the encoding of other characters, but that's not the case of all. For instance, in practice, the encoding of `x` is found at the end of the encoding of many characters in the BIG5 or GB18030 encodings, so if a `x` was appended instead of `.`, that `x` could end up being combined with another byte that happens to be present at the end of the output to form a new character and `${var%x}` would then fail to remove that `x`.}

Answer 2

The only shells that I know that have a command substitution that can be told not to remove trailing newline characters are rc (the shell of Unix V10 and plan9, and Byron Rakitzis's public domain clone and its derivatives such as es or akanga) and fish.

In rc-like shells,

var = `{somecommand}

Stores the $ifs delimited words in the output of somecommand into the $var variables (and variables are arrays/lists in rc).

With an empty $ifs, (ifs = ()), that stores the output as is. The separators can also be specified with the ``(separators){somecommand}, so:

var = ``(){somecommand}

But rc/es/akanga are not very Bourne-like.

Also, you generally want to remove one newline character.

For instance in:

dirname=$(dirname -- "$file")

You do want to remove the newline character added by dirname, but not the ones that may be at the end of $file's directory.

rc has no builtin operator for that. You'd need:

dirname = ``(){dirname -- $file | head -c -1}

(not in standard head). In es, you can use the ~~ pattern extraction operator:

nl = '
'
dirname = <={ ~~ ``(){dirname -- $file} *$nl }

Which is hardly more legible (though could be made into a function).

The fish shell command substitution splits the output into its constituent lines.

set var (somecommand)

Stores the lines of somecommand's output into the $var array (and empty lines are preserved even those at the end of the output).

If you set $IFS to the empty list or one empty string, that splitting is disabled and up to one newline character is removed from the end of the output (at least in current versions of fish, IIRC there have been several changes on that front over the years). So there,

begin; set -l IFS; set dir (dirname -- $file); end

is reliable.

I'm not aware of any Bourne-like shell that can be told not to strip all trailing newline characters. That behaviour is required by POSIX and all Bourne-like shells do it even when not in posix mode (for those that have one like bash or zsh).

In bash 4.4+, in place of command substitution, you can also use readarray combined with process substitution (or a pipe with the lastpipe option and in non-interactive instances):

readarray -td '' var < <(somecommand)

Here -d '' is to splits on NULs. bash doesn't support storing NUL in its variables anyway, so for a replacement of command substitution that would be enough.

readarray -t lines < <(somecommand)

Stores that lines of the output of somecommand into the $lines array. You could then join them with newline to reconstruct that output without one trailine newline:

IFS=$'\n'
output="${lines[*]}"

But in all those approaches, you lose the exit status of somecommand.

To work around that misfeature of command substitution removing all newline characters, a common idiom is to add a non-newline character and stripping it (along with one newline character) afterwards as Gordon said.

But to do it without losing the exist status, you'd so something like:

cmdsubst() { # args: var cmd args
  eval "$1"'=$(shift; "$@"; ret=$?; printf .; exit "$ret")'
  eval "$1=\${$1%.}; $1=\${$1%'
'}; return $?"
}

Which is like command substitution except that only up to one newline character is removed.

To be used as:

cmdsubst dir dirname -- "$file"

in place of:

dir=$(dirname -- "$file")

Or for more complex commands:

cmdsubst var eval 'cmd1; for i in a b; do cmd "$i"; done'

Answer 3

A simple way that works in many use cases is to surround the command substitution with double quotes, and insert a newline immediately prior to the closing quote. By placing the newline inside the quotes but outside the parentheses, you remove it from the scope of the command substitution, and the shell won't automatically remove it.

The first example has no newline before the closing quote, so the next $ prompt appears immediately after the output. The second example works.

$ printf '%s' "$(date)"
Mon Jan 22 13:39:23 PST 2024$ printf '%s' "$(date) 
"
Mon Jan 22 13:39:26 PST 2024
$

Because the shell strips all trailing newlines within the parenthesized command substitution, and the script subsequently adds just one newline back, this method has the limitation that it preserves only one trailing newline. If there are multiple newlines, they are condensed down to one.