QUESTION
In bash you can pass data from step to step by piping data:
program1 | program2 | program3 | ...
Or you can do this with variables (depending how each program work):
RES1=$(program1)
RES2=$(program2 $RES1)
...
I am looking for counterpart of such passing data in parallel.
EXAMPLE
I try to convert my "linear" script to the amazing parallel (this tool still do wonders for me :-D), but I have a problem with reading and using the output from the first step. Original version:
for fn in $(ls $REV *)
do
DATA=$(sh script1.sh ${fn})
sh script2.sh $DATA
done
Here how I tried to do this with parallel
:
ls $REV * | parallel -j+0 DATA=$(sh script1.sh {}) \; sh script2.sh $DATA
However with this code myscript
gets some broken data on input. Version with removed reading the result and second step altogether, works:
ls $REV * | parallel -j+0 sh script1.sh {}
So how to read the output from the first step of parallel
and use it in next steps?
PROBLEM
For easier debugging let's say my first script (script1.sh) is:
echo "RECEIVED THIS ${1}"
And the main script is:
ls * | parallel -j+0 RES="$(sh script1.sh {})"
(I skipped the script2 here for testing capturing the output of script1). Then the result of entire execution is this:
/bin/bash: THIS: command not found
/bin/bash: THIS: command not found
/bin/bash: THIS: command not found
...
script1.sh
? What aboutscript2.sh
? And what command did you run to produce this output? – Gilles 'SO- stop being evil' Jul 27 '15 at 07:32script2
to test (it can be empty file) because the problem is withscript1
. I cannot read anything fromscript1
, because when I run it with capture (i.e.OUTPUT=$(sh script1)
) I get the error as shown in my post. In other words I can runscript1
in non-capture mode, but not in capture mode. Because of this problem I didn't test further steps, because it didn't make sense. – greenoldman Jul 27 '15 at 07:59RES=....
. How do you merge 4 steps using first approach? One script solves the problem, but in such way, that I had to have plethora of scripts, just for connecting one program to another. This can be used for single case, but not as general solution. As I wrote, I am looking for counterpart of bash piping and/or capturing the output. – greenoldman Jul 27 '15 at 09:17myfunc() { DATA=$(sh script1.sh $1); sh script2.sh $DATA }; export -f myfunc; ls $REV * | parallel myfunc
– Ole Tange Jul 28 '15 at 22:06