2

Right now I have this:

 echo "$run_test" | bash 2>&1 | prepend "r2g-test:" "yellow";

What this does is prepend "r2g-test:" to each line of the stdout/stderr. Because I am sending stderr to stdout, the prepend program doesn't know the difference because it's all stdin to it.

Is there some way I can send stderr to a different prepend instance, perhaps with tee?

Perhaps something like this?

 echo "$run_test" | bash 2> $(prepend 'r2g-test:' 'red') | prepend 'r2g-test:' 'yellow';

This might work, with process substitution:

bash 2> >(prepend 'r2g-test:' 'red')

But so far, the stderr never shows up in the terminal

Alexander Mills
  • 10,002

1 Answers1

3

First, let's create a program run_test which generates both stdout and stderr:

$ run_test() { while sleep 0.2; do echo "Out $((++c))"; echo "err$c">&2; done; }

Now, let's send the stdout and stderr to different filters. Since I don't have a prepend installed, I will use sed for the same purpose:

$ exec 3>&2; { run_test | sed 's/^/stdout: /'; } 2>&1 1>&3 | sed 's/^/stderr: /'
stdout: Out 1
stderr: err1
stdout: Out 2
stderr: err2
stdout: Out 3
stderr: err3

How it works

  • exec 3>&2

    This creates file descriptor 3 as a duplicate of stderr.

  • run_test | sed 's/^/stdout: /'

    This runs run_test and prepends stdout: to the beginning of stdout.

  • { run_test | sed 's/^/stdout: /'; } 2>&1 1>&3

    2>& redirects stderr to stdout so that stderr will go into the next pipe. 1>&3 redirects stdout to stderr so that it appears on the terminal.

  • { run_test | sed 's/^/stdout: /'; } 2>&1 1>&3 | sed 's/^/stderr: /'

    The last pipe captures run_test's stderr, (which is now stdout) and prepends stderr: to it.

Using process substitution

$ run_test > >(sed 's/^/stdout: /') 2> >(stdbuf -oL sed 's/^/stderr: /' >&2)
stdout: Out 1
stderr: err1
stdout: Out 2
stderr: err2
stdout: Out 3
stderr: err3

The above uses stdbuf which is standard on Linux. For other OS's, one will need to look for analogous commands.

John1024
  • 74,655
  • thanks! There must be a simpler way! I saw something like this using an intermediate fd, but I hoping for something less xrazy. What about this: echo "$run_test" | bash 2> >(prepend 'r2g-test:' 'red') | prepend 'r2g-test:' 'yellow'; ... why won't that work? – Alexander Mills May 07 '18 at 07:51
  • that uses process substitution, but the stderr never seems to show up in the terminal – Alexander Mills May 07 '18 at 07:52
  • 1
    @AlexanderMills I added a solution using process substitution. Note that anytime one one separates the two streams, output order is not guaranteed. – John1024 May 07 '18 at 08:00
  • thanks, I asked a new question, because this one was a bit unclear: https://unix.stackexchange.com/questions/442250/stderr-is-being-sent-down-pipeline-but-i-dont-want-that – Alexander Mills May 07 '18 at 08:07
  • nice sed prepending trick by the way, i didn't know that you could do that – Alexander Mills May 07 '18 at 08:08
  • I guess I don't understand what this is for: – Alexander Mills May 07 '18 at 08:09
  • run_test > >(sed 's/^/stdout: /') 2> >(stdbuf -oL sed 's/^/stderr: /' >&2) – Alexander Mills May 07 '18 at 08:09
  • what is >&2 for in the above command? – Alexander Mills May 07 '18 at 08:09
  • >&2 sends stdout to stderr for that process. Without that, the stderr gets captured again by stdout resulting in lines like stdout: stderr: err1. – John1024 May 07 '18 at 08:11
  • Got it thanks. Using your first technique with exec 3>&2, looks like stderr and stdout are not ordered. Is there some way to force them to always appear in the order in which they were produced? They are produced by the same bash process, so nothing out of the ordinary. – Alexander Mills May 07 '18 at 08:22
  • this is what I have for context: exec 3>&2; { echo "$run_test" | bash | prepend 'r2g-test: ' 'yellow'; } 2>&1 1>&3 | prepend 'r2g-test: ' 'red' – Alexander Mills May 07 '18 at 08:24
  • the stderr is appearing first, when it should be appearing second in my case – Alexander Mills May 07 '18 at 08:26
  • Anytime that the two streams are separated, output order is not guaranteed. It depends on buffering for which stdbuf -oL might help. Also, there are three processes running: bash and the two prepends. The final output order also depends on which prepend happens, under the vagaries of multitasking, to get more CPU time than the other. – John1024 May 07 '18 at 08:28
  • well what if you could separate them and then reconnect them? would that help to guarantee a little more order than o/w? prolly right – Alexander Mills May 07 '18 at 08:29