If I understood well all your requirements you could achieve that in bash by creating an unnamed pipe per command, then redirecting each command’s output to its respective unnamed pipe, and finally retrieving each output from its pipe into a separate variable.
As such, the solution might be like:
: {pipe2}<> <(:)
: {pipe3}<> <(:)
command1 | tee >({ command2 ; echo EOF ; } >&${pipe2}) >({ command3 ; echo EOF ; } >&${pipe3}) > /dev/null &
var2=$(while read -ru ${pipe2} line ; do [ "${line}" = EOF ] && break ; echo "${line}" ; done)
var3=$(while read -ru ${pipe3} line ; do [ "${line}" = EOF ] && break ; echo "${line}" ; done)
exec {pipe2}<&- {pipe3}<&-
Here note particularly:
- the use of the
<(:) construct; this is an undocumented Bash's trick to open "unnamed" pipes
- the use of a simple
echo EOF as a way to notify the while loops that no more output will come. This is necessary because it's no use to just close the unnamed pipes (which would normally end any while read loop) because those pipes are bidirectional, ie used for both writing and reading. I know no way to open (or convert) them into the usual couple of file-descriptors one being the read-end and the other its write-end.
In this example I used a pure-bash approach (beside the use of tee) to better clarify the basic algorithm that is required by the use of these unnamed pipes, but you could do the two assignments with a couple of sed in place of the while loops, as in var2="$(sed -ne '/^EOF$/q;p' <&${pipe2})" for variable2 and its respective for variable3, yielding the same result with quite less typing. That is, the whole thing would be:
Lean solution for small amount of data
: {pipe2}<> <(:)
: {pipe3}<> <(:)
command1 | tee >({ command2 ; echo EOF ; } >&${pipe2}) >({ command3 ; echo EOF ; } >&${pipe3}) > /dev/null &
var2="$(sed -ne '/^EOF$/q;p' <&${pipe2})"
var3="$(sed -ne '/^EOF$/q;p' <&${pipe3})"
exec {pipe2}<&- {pipe3}<&-
In order to display the destination variables, remember to disable word splitting by clearing IFS, like this:
IFS=
echo "${var2}"
echo "${var3}"
otherwise you’d lose newlines on output.
The above does look quite a clean solution indeed. Unfortunately it can only work for not-too-much output, and here your mileage may vary: on my tests I hit problems on around 530k of output. If you are within the (well very conservative) limit of 4k you should be all right.
The reason for that limit lies to the fact that two assignments like those, ie command substitution syntax, are synchronous operations, which means that the second assignment runs only after the first is finished, while on the contrary the tee feeds both commands simultaneously and blocking all of them if any happens to fill its receiving buffer. A deadlock.
The solution for this requires a slightly more complex script, in order to empty both buffers simultaneously. To this end, a while loop over the two pipes would come in handy.
A more standard solution for any amount of data
A more standard Bashism is like:
declare -a var2 var3
while read -r line ; do
case "${line}" in
cmd2:*) var2+=("${line#cmd2:}") ;;
cmd3:*) var3+=("${line#cmd3:}") ;;
esac
done < <(
command1 | tee >(command2 | stdbuf -oL sed -re 's/^/cmd2:/') >(command3 | stdbuf -oL sed -re 's/^/cmd3:/') > /dev/null
)
Here you multiplex the lines from both commands onto the single standard “stdout” file-descriptor, and then subsequently demultiplex that merged output onto each respective variable.
Note particularly:
- the use of indexed arrays as destination variables: this is because just appending to a normal variable becomes horribly slow in presence of lots of output
- the use of
sed commands to prepend each output line with the strings "cmd2:" or "cmd3:" (respectively) for the script to know which variable each line belongs to
- the necessary use of
stdbuf -oL to set line-buffering for commands’ output: this is because the two commands here share the same output file-descriptor, and as such they would easily override each other’s output in the most typical race condition if they happen to stream out data at the same time; line-buffering output helps avoiding that
- note also that such use of stdbuf is only required for the last command of each chain, ie the one outputting directly to the shared file-descriptor, which in this case are the sed commands that prepend each commandX’s output with its distinguishing prefix
One safe way to properly display such indexed arrays can be like this:
for ((i = 0; i < ${#var2[*]}; i++)) ; do
echo "${var2[$i]}"
done
Of course you can also just use "${var2[*]}" as in:
echo "${var2[*]}"
but that is not very efficient when there are many lines.
&). – Archemar Mar 19 '19 at 08:08