How to run bash script via multithreading

Question

I have bash script which creates high energy collision events, if one event takes around 2 minutes then for 100000 events it takes 200000 minutes, which is too much. so if I have 50 cores in a node, I want to have one event on each core, so in this way there will be 50 events generated in 2 minutes, so time will be saved. I thought this can be done by multi-threading, or any other way by which this can be done?, so can anyone help me?

Answer 1

GNU Parallel is made for this kind of tasks.

seq 100000 | parallel do_experiment

If your experiment takes different kind of values (say, models), then you can run all experiments for all models by:

seq 100000 | parallel do_experiment --iteration {1} --model {2} :::: - ::: model1 model2

It will default to 1 process per CPU core and it will make sure the output of two parallel jobs will not be mixed. If you have multiple computers sitting unused in your lab, you can make them take part in the calculation, too.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Answer 2

You can start the jobs as background processes untill you reach the limit of 50 jobs in background, then wait for a job to finish before starting another one.

LIMIT=50
while collision_event_to_run
do 
    run_new_collision_event&

    while (( $(jobs | wc -l) >= LIMIT ))
    do
                sleep 1
    done

done

or better to run new events while the load average is low.

LIMIT=49
while collision_event_to_run
do 
    run_new_collision_event&
    sleep 2   # let time for the load average calculation

    while (( $(cut -d " " -f1 < /proc/loadavg) >= LIMIT ))
    do
                sleep 1
    done

done

Another option is to use the batch command to stack the jobs in the atd batch queue.

batch collision _event_1
batch collision _event_2
batch collision _event_3
...

atd service will starts the job in parallel until the server load reach a limit. That limit is a parameter of atd, it must be set in the atd service starting script, atd -l 50 for instance.

Edit:

• To keep a little free cpu for the operating system atd -l 49.

• atd will start a new job every minute, which is too slow, to permit the server to reach the load as jobs last 2mn. You can reduce the delay with the -b parameter. atd -b 2 -l 49 to start a job every 2 seconds will permit to reach the limit before the first jobs finishes.