Parallelizing a loop in Bash

Question

I have a Bash script in which there is a loop to enter a specific directory and it does a certain calculation on a specific input file. After that, it exits from the directory and does the same thing within another directory with a different input file. The calculation needs a lot of time and I would like to parallelize it.

How I can modify my script? Is there an option an option to do that?

myscript.sh

cd MainDir
for dir in *
  do
      cd ${dir}
      LD_LIBRARY_PATH="$software"/ "$software"/calc -i /home/files/"$dir.txt" -l /home/Str/Art.pdb -a 5.0 -rf /home/file/prot -cpu 1 opt -w ${dir}_res > ${dir}_WPA.log
     cd .. 
  done

I use the command -cpu to indicate how cpu use. I have many CPUs to use so how can I parallelize more jobs?

e.g If I have three different input files I would like to run (in adifferent directory) together the following command:

cd 1
LD_LIBRARY_PATH="$software"/ "$software"/calc -i /home/files/1.txt -l /home/Str/Art.pdb -a 5.0 -rf /home/file/prot -cpu 1 opt -w 1_res > 1_WPA.log
-----------
cd 2
LD_LIBRARY_PATH="$software"/ "$software"/calc -i /home/files/2.txt -l /home/Str/Art.pdb -a 5.0 -rf /home/file/prot -cpu 1 opt -w 2_res > 2_WPA.log
-----------
cd 3
LD_LIBRARY_PATH="$software"/ "$software"/calc -i /home/files/3.txt -l /home/Str/Art.pdb -a 5.0 -rf /home/file/prot -cpu 1 opt -w 3_res > 3_WPA.log

Could someone help me please? Thanks.

Answer 1

You can add an & at the end of the command to send it to the background:

for i in 1 2 3 4; do
    (
        cd dir
        command
        [...]
    ) &
done
wait # pause until all background processes are terminated

Answer 2

With zsh:

do-calc() (
  cd -- $1 &&
    LD_LIBRARY_PATH=$software/ $software/calc \
      -i /home/files/$1.txt \
      -l /home/Str/Art.pdb \
      -a 5.0 \
      -rf /home/file/prot -cpu 1 opt -w ${1}_res > ${1}_WPA.log
)
autoload zargs
cd MainDir && zargs -rn1 -P12 -- ./*(N-/) -- do-calc

To run up to 12 of those do-calc functions in parallel.

With any Korn-like shell with process substitution support (such as bash -- the GNU shell) and with GNU utilities, you could do something similar with:

export software
cd MainDir &&
  xargs -0rn1 -P12 -a <(
      LC_ALL=C find . -maxdepth 1 ! -name '.*' -xtype d -print0 |
        sort -z
    ) sh -c '
      cd -- "$1" &&
        LD_LIBRARY_PATH="$software/" "$software/calc" \
          -i "/home/files/$1.txt" \
          -l /home/Str/Art.pdb \
          -a 5.0 \
          -rf /home/file/prot -cpu 1 opt -w "${1}_res" > "${1}_WPA.log"
      ' sh

Answer 3

With GNU Parallel you would do something like:

doit() {
      dir="$1"
      cd ${dir}
      LD_LIBRARY_PATH="$software"/ "$software"/calc -i /home/files/"$dir.txt" -l /home/Str/Art.pdb -a 5.0 -rf /home/file/prot -cpu 1 opt -w ${dir}_res > ${dir}_WPA.log
}
export -f doit
cd MainDir
parallel doit  ::: *

This will run one job per CPU thread. If you do not like that, you can adjust it to running 13 jobs in parallel with:

parallel -j13 doit  ::: *

If you have files like ':::' you need to do something like:

LC_ALL=C find . -maxdepth 1 ! -name '.*' -xtype d -print0 |
  parallel -0 doit

Or:

parallel --argsep /// -j13 doit  /// *