How can sed replacement string be a command's output?

Question

Below sed command helps add text Hello to the end of each line of the input

<complex_query> | sed "s,$,Hello,"
Output:
myvar1: Hello
myvar2: Hello

I now wish to have the output of grep wow data.txt | cut -d: -f2 in place of Hello

How is that possible?

Below is something i tried but does not work.

<complex_query> | sed "s,$,(grep wow data.txt  | cut -d: -f2),"

Expected Desired output:

myvar1: <output of grep wow data.txt  | cut -d: -f2 command>
myvar2: <output of grep wow data.txt  | cut -d: -f2 command>

Answer 1

To use the output of a command (minus trailing newline characters) in an argument of another command, you'd use command substitution, a feature introduced by the Bourne shell in the late 70s.

In the Bourne shell, the syntax was `cmd...`. So was it in csh.

In modern sh and sh-like shells such as ksh (where it comes from)¹, zsh, bash, ash-based shells, the syntax is $(...) instead (though they do also support the Bourne syntax for backward compatibility).

In rc-like shells, the syntax is `{cmd...}

In the fish shell, it's (cmd ...) though that can't be used inside double quotes. Since version 3.4.0, fish also supports $(cmd...) which can be used inside double quotes.

Command substitution generally splits the output of the command in ways that vary between shells resulting in several arguments passed to the commands, some (including most Bourne-like shells) even perform globbing on the resulting words (!). That can usually be prevented by placing the command substitution inside double quotes (not in rc-like shells² nor in fish < 3.4.0³).

So here:

• POSIX-like shells or fish 3.4.0+:

  <complex_query> | sed "s,\$,$(grep wow data.txt | cut -d: -f2),"
  

• rc-like shells:

  nl = '
  '
  <complex_query> | sed 's,$,'``($nl){grep wow data.txt | cut -d: -f2},
  

• csh/tcsh:

  <complex_query> | sed "s,\$,`grep wow data.txt | cut -d: -f2`,"
  

• fish < 3.4.0

  <complex_query> | sed 's,$,'(grep wow data.txt | cut -d: -f2 | string collect),
  

Now, beware the & and \ characters are special in the replacement part of sed's s command, so would have to be escaped with \ along with , (the separator used here) and newline if present there (if newline can be present, beware of splitting in csh/rc as noted above).

See How to ensure that string interpolated into `sed` substitution escapes all metachars about that.

Another approach is to use perl instead of sed to not have to worry about special characters:

<complex_query> | REPL="$(cmd...)" perl -lpe '$_ .= $ENV{REPL}'

(here assuming a POSIX-like shell)

Or even let perl collect cmd's output itself (using its own `...`, and trimming one trailing newline if any with chomp).

<complex_query> | perl -lpe 'BEGIN{chomp ($repl = `cmd...`)}; $_ .= $repl'

---

^{¹ ksh93 and recent versions of mksh also support a ${ ...; } form which unlike the other one, but like in fish doesn't introduce a subshell environment.}

^{² rc-like shells have only one type of quotes: '...' which are strong quotes (no expansion can happen within). You'd prevent the splitting by using ``(){cmd ...} (using the ``(sep1 sep2){cmd ...} syntax where you specify the list of separators for splitting, here empty) though that would also prevent trailing newline trimming.}

^{³ also note that in csh/tcsh, "`cmd ...`" still splits on newlines.}