2

I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files in parallel (about 1000). What I do now, is put all the commands

./pgm -a file1 -b 12 > out1.txt &
./pgm -a file2 -b 14 > out2.txt &
./pgm -a file3 -b 16 > out3.txt &
./pgm -a file4 -b 18 > out4.txt &
...

in a file, and execute this file. I thought this would use all the available processors, but the number of parallel execution is very limited.

How can I achieve this? Note that parallel command is not an option.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
wwjoze
  • 121
  • What machine is this? Is this a normal PC or some kind of server with multiple nodes? – terdon Feb 18 '14 at 15:01
  • Maybe pgm is i/o bound then extra CPU doesn't help. vmstat 1 1 in a second terminal whilst other is running for a few secconds would give you a rough idea. – X Tian Feb 18 '14 at 15:03
  • 1
    Which parallel are you referring to GNU parallel or the one from moreutils? Or both? – Anthon Feb 18 '14 at 15:57
  • Can you based on http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html elaborate why GNU Parallel is not an option? – Ole Tange Feb 18 '14 at 19:48
  • Thanks for the answer. @OleTange: I tried but the installation failed, and I am not root. – wwjoze Feb 19 '14 at 00:10
  • @XTian: you are right, there are IO blockings. vmstat gives bo=1. Does this explains why when I run xargswith -P8 there is only one process running sometime? – wwjoze Feb 19 '14 at 00:11
  • vmstat 1 1 is not useful because it gives you the average statistics since the system booted. vmstat 3 (or iostat -z 3) to get running statistics every 3 seconds. Did you add the -n1 to xargs. You don't have to install parallel system-wide to use it, you can put it somewhere in your home directory. – Stéphane Chazelas Feb 19 '14 at 06:53
  • @wwjoze Can you elaborate how '(wget pi.dk/3 -qO - || curl pi.dk/3/) | bash' fails? What error do you get? – Ole Tange Feb 19 '14 at 13:46

2 Answers2

5

With GNU xargs:

seq 1000 | xargs -P4 -n1  sh -c 'exec ./pgm -a "file$1" -b 12 > "out.$1"' sh &

Would run up to 4 ./pgms in parallel.

Otherwise, with pdksh/mksh/oksh:

trap : CHLD
n=0
for f in file*; do
  jobs=$(jobs | wc -l)
  if (($jobs < 4)); then
    ./pgm "$f" > out.$((++n)) &
  else
    wait
  fi
done
trap - CHLD
wait

the details of signal handling vary from one shell to the next. That trick works in pdksh and its derivatives but not in any other shell I tried. You need a shell where one can trap SIGCHLD (excludes bash), where the SIGCHLD handler is executed straight away (not blocked during a wait) (excludes ash, yash), where the SIGCHLD handling interrupts the wait (excludes ksh93 and zsh).

In shells other than bash, you could also look at approaches where jobs are started in the SIGCHLD handler.

Stéphane Chazelas
  • 544,893
  • Thanks for the command line using xargs. But it seems my process uses to much IO to be fully parallelized. – wwjoze Feb 19 '14 at 00:22
1

I was having a similar kind of issue. Since you had specified parallel is not an option, you might need to look into swift as discussed in the answer of this question.

Community
  • 1
Ramesh
  • 39,297