Alternative to wait -n (because server has old version of bash)

Question

I would like to solve the following issue about submitting a job that has been parallelised to a specific node.

Let me start with explaining the structure of my problem

I have two very simple Matlab scripts

1) main.m

clear
rng default
P=2;
grid=randn(4,3);
jobs=1;

2) f.m

sgetasknum_grid=grid(jobs*(str2double(getenv('SGE_TASK_ID'))-1)+1: str2double(getenv('SGE_TASK_ID'))*jobs,:); %jobsx3

result=sgetasknum_grid+1; 

filename = sprintf('result.%d.mat', ID);
save(filename, 'result')

exit

What I want to do is:

Run main.m;
then, run f.m 4 times, allowing for parallel execution of 2 tasks at each time
Everything should be executed on node A

Here's how I implement the steps above

1) I save main.m and f.m into a folder named My_folder

2) I create the script td.sh as below and save it into the folder My_folder

#!/bin/bash -l
#$ -S /bin/bash
#$ -l h_vmem=5G
#$ -l tmem=5G
#$ -l h_rt=480:0:0
#$ -cwd
#$ -j y


#$ -N try

date
hostname

J=4 #number tasks

N=2 #number tasks executed in parallel

export SGE_TASK_ID


SGE_TASK_ID=1
n=0
while [ "$SGE_TASK_ID" -le "$J" ]; do
    if [ "$n" -eq "$N" ]; then
        wait -n  # as soon as one task is done, refill it with another
        n=$(( n - 1 ))
    fi

    printf 'Task ID is %d\n' "$SGE_TASK_ID"

    /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash -r "main; ID=$SGE_TASK_ID; f; exit" &

    SGE_TASK_ID=$(( SGE_TASK_ID + 1 ))
    n=$(( n + 1 ))
done

wait

3) I go into the terminal and type ssh username@A, then cd /.../My_folder, then bash td.sh

Problem: I get the following error

td.sh: line 26: wait: -n: invalid option
wait: usage: wait [id]

As noticed in the comments below, the issue is that the version of bash on @A is old (the -n option was added to the wait builtin in 4.3) and the sysadmin can't update it. The latest version possible is bash 4.1.

Thus, could you suggest a way to replace wait -n?

Perhaps the version of bash on the remote node is older? IIRC the -n option was added to the wait builtin in 4.3 — steeldriver, Dec 07 '18 at 13:11
OK. What can I do? (1) contact sysadmin, (2) replace wait -n with? — Star, Dec 07 '18 at 13:15
Before making the code more complicated, see if bash 4.3 or newer are installed elsewhere on the system, or if an admin is happy to upgrade the existing bash to a later version. — Kusalananda, Dec 07 '18 at 13:17
Thanks. The sysadmin said that the latest version possible is bash 4.1 and that in order to get a newer version he would need to upgrade the entire OS. What can I do to solve the issue? — Star, Dec 10 '18 at 09:52
If you remove all the stuff about matlab, and ask the question at the top, then you may get an answer. At present people are starting to read, saying to themselfs “I don't know about matlab”, and moving on. Many of these people may know the answer, but never get to the question. — ctrl-alt-delor, Dec 11 '18 at 13:18
Note that first line of you script #!… is magic. If you do chmod +b td.sh then you can run it as a normal executable e.g. ./my.sh. And remove the file extension, as when you have to re-implement in python, you don't want to have to rename it. — ctrl-alt-delor, Dec 11 '18 at 13:21
Is switching to another shell (like zsh, dash, mksh, ksh93) where it's easier an option? — Stéphane Chazelas, Dec 11 '18 at 13:24
@StéphaneChazelas I have no idea, if you tell me how to do it I can try. — Star, Dec 11 '18 at 18:07

ctrl-alt-delor · Answer 1 · 2018-12-11T13:51:51.033

That script you have written can better be done with gnu parallel, or make with the -j option. Alternatively you can re-write it in python (or another language).

Look at

parallel: A tool for use in bash (the easiest of the 3 to learn, only does one thing).
make: A bit more advanced, and it has its own language. It is used to create files. e.g. to make A.b you will need A.a, and g.f, when you have these, do z;y;z. You can also add rules on how to make A.a and g.f. It will work out what depends on what, and build things in the correct order. If it can it will do things in parallel (if asked to).
python: A programming language, it can do what you script is trying to do, it can do what matlab does.

You will also have to consider which of these are/can-be installed. Do this to find out:

type parallel
type make
type python

Note: type is not an instruction to you, to type. It is the command, that you type. It tells you the type of each command (where it is).

score 1 · Accepted Answer · answered Dec 10 '18 at 22:40

1

What about not using wait at all, in the while loop?

while [ "$SGE_TASK_ID" -le "$J" ]; do

    # grep count of matlab processes out of list of user processes
    n = $(ps ux | grep -c "matlab")

    ##  if [ "$n" -le "$N" ]; then
    if [ "$n" -eq "$N" ]; then
        # sleep 1 sec if already max processes started
        sleep 1
        ##  wait -n  # as soon as one task is done, refill it with another
        ##  n=$(( n - 1 ))
    else
        # start another process
        printf 'Task ID is %d\n' "$SGE_TASK_ID"

        /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash -r "main; ID=$SGE_TASK_ID; f; exit" &

        SGE_TASK_ID=$(( SGE_TASK_ID + 1 ))

    fi
    ##  n=$(( n + 1 ))
done

The string to grep for may of course have to differ, depending of what you have running (e.g. give f.m some more special name, and grep for that.)

answered Dec 10 '18 at 22:40

Jaleks

2,579

Thanks. (1) Sorry, but I don't understand the terminology: what do you mean by "The string to grep for may of course have to differ, depending of what you have running (e.g. give f.m some more special name, and grep for that.) (2) Should SGE_TASK_ID=1 n=0 be inserted before while? (3) Should I out the wait after the final done? – Star Dec 11 '18 at 12:08
n = $(ps ux | grep -c "matlab") counts the Matlab processes used by other users in the node. Why are you counting just the Matlab processes and not more generally any process? Is that line aiming to check how many processes are available in the node out of the N maximum number of processes? In that case, wouldn't be better something like n=$(cut -d. -f1 /proc/loadavg) (which counts how many processes are used by me or other users)?

Star

Dec 11 '18 at 18:02

What sleep 1 is exactly doing? Is it "waiting one second and then recomputing n"?

– Star Dec 11 '18 at 18:06

score 1 · Answer 3 · answered Dec 12 '18 at 14:52

1

Your shell script looks suspiciously like it's written for a distributed resource manager (probably gridengine), which supports running multiple things in parallel out of the box, with no shell scripting required. Why not use those features?

qsub -t 1-4 ./script.sh

then remove all lines that modify the value of SGE_TASK_ID -- gridengine will set that for you.

(you can also set the option in the script, using a line like #$-t 1-4 if you prefer, of course...)

answered Dec 12 '18 at 14:52

Wouter Verhelst

9,331

Thanks, but I thought that qsub does not run all things in node A (as instead I want). Am I right? – Star Dec 12 '18 at 14:56
It does not do so by default, no, but it is possible to add parameters to make it do so. How exactly you do that depends on what you need to do; you can tell it "please run all these jobs in parallel, but make them run on the same node" (use -pe to select a parallel environment that is configured like that, usually called smp), or "please run all these jobs whenever, but make sure they all run on explicitly this host (select a queue with @hostname at the end -- -q foo@A). If you need more details, I would suggest you ask another question, because this is something entirely different... – Wouter Verhelst Dec 12 '18 at 15:58
Thanks. I'm a beginner and hence I'm about to ask something probably very silly: what I care at the moment is just speed of execution of my Matlab functions main.m and f.m; is my script causing main.m and f.m to be executed slower than if I used your method? If not, why your method is preferable? – Star Dec 12 '18 at 16:03
@user3285148 it wouldn't necessarily be much slower; depending on scheduling and on how long your scripts run, it might even be faster. But it's much easier to do, since you don't need to do any shell scripting :-) – Wouter Verhelst Dec 13 '18 at 13:01

score 1 · Answer 4 · edited Dec 16 '18 at 20:11

1

Use GNU Parallel.

Newer versions have --embed which will embed GNU Parallel into a shell script. This way you not need to have GNU Parallel installed on the cluster.

So on your laptop install the newest version of GNU Parallel and do:

$ parallel --embed > myscript.sh

Now edit myscript.sh:

#!/bin/bash -l
#$ -S /bin/bash
#$ -l h_vmem=5G
#$ -l tmem=5G
#$ -l h_rt=480:0:0
#$ -cwd
#$ -j y


#$ -N try

# Here starts the original content of myscript.sh
# Embedded GNU Parallel created with --embed
parallel() {
   «This bit removed for brevity (around 13000 lines, generated by gnu parallel)»
   return `cat "$_exit_FILE"; rm "$_exit_FILE"`
}
# Here ends the original content of myscript.sh

date
hostname

J=4 #number tasks
N=2 #number tasks executed in parallel

doit() {
    SGE_TASK_ID="$1"
    printf 'Task ID is %d\n' "$SGE_TASK_ID"

    /share/.../matlab -nodisplay -nodesktop -nojvm -nosplash -r "main; ID=$SGE_TASK_ID; f; exit"
}
export -f doit

seq $J | parallel -j $N doit

Finally copy myscript.sh to the server the same place you have td.sh and run it like you run td.sh.

edited Dec 16 '18 at 20:11

ctrl-alt-delor

27,993

answered Dec 12 '18 at 22:40

Ole Tange

35,514

Thanks: (1) the HPC of my university does not have parallel installed: can I install it or do I need to contact the sysadmin? (2) Where is SGE_TASK_ID in your script? I need it because it is used in f.m (3) When you say "Finally copy myscript.sh to the server" you mean save myscript.sh into the folder where I'm cd? (4) Which line is used to execute myscript.sh into the node A? – Star Dec 13 '18 at 08:43
See the edited changes. – Ole Tange Dec 13 '18 at 13:42
Thanks. It's still unclear to me why I should install the newest version of GNU Parallel in MY LAPTOP and do $ parallel --embed > myscript.sh (in the terminal of my laptop I guess?). I run all my staff by remotely accessing to the HPC. Why is my laptop involved? – Star Dec 13 '18 at 13:49
1

Your laptop is involved because I assume you can install GNU Parallel on your laptop without asking the sysadmin. It does not need to be your laptop - it can be any machine that has a recent version of GNU Parallel. By doing this you do not need to ask your sysadmin to install GNU Parallel on the HPC. – Ole Tange Dec 13 '18 at 13:51

score 0 · Answer 5 · edited Dec 11 '18 at 13:52

0

You can install it yourself:

For most software, the only part of the installation process that needs root privileges is to copy files into the standard places. (not true for device-drivers, kernels, virtual machines, etc.)

I advise to install Bash into your home directory (~/bin) and get your script to use it.

edited Dec 11 '18 at 13:52

Vlastimil Burián

28,462

answered Dec 11 '18 at 13:12

ctrl-alt-delor

27,993

Thanks a lot but your language/advice is too high level for me. I'm just a beginner, coming from a completely different field. I need to be guided step by step at this stage. Sorry. – Star Dec 11 '18 at 13:27

score 0 · Answer 6 · answered Dec 15 '18 at 05:08

The question says, basically

What I want to do is: Run main.m; then, run f.m 4 times, allowing for parallel execution of 2 tasks at each time.

It becomes apparent from your script that you want f.m to be passed a sequential argument of "1", "2", "3" and "4". As far as I can see, the rest is inconsequential detail.

Here's a simple way of doing that:

main.m
(f.m 1; f.m 3)&
(f.m 2; f.m 4)&
wait

It will clearly run f.m the desired four times, and wait for them all to complete. Two processes will be running simultaneously (in parallel), because the second compound command — i.e., (f.m 2; f.m 4) — will start running immediately after the first one — i.e., (f.m 1; f.m 3) — has started. And a maximum of two processes will be running simultaneously (in parallel), because each of the two compound commands runs only one f.m process at a time.

This might not result in optimum scheduling. For example, if f.m 1 and f.m 3 each takes a minute to run, and f.m 2 and f.m 4 each takes an hour to run, then the above will take two hours to run — and most of that time, there will be only one task running — rather than one hour and one minute, which many of the other attempts here attempt to do. But the question doesn't specify optimum scheduling, and your f.m doesn't look like different invocations of it would take wildly different times. (Then again, I don't know Matlab, so I don't know what f.m is doing.)

The question doesn't ask for a generalized solution, where the 2 and the 4 are parameters, but the sample code is generalized, and so are most of the other answers, so here's a generalized version of the above:

run_f()
{
        # Usage: run_f start_index step max
        local j

        for ((j=$1; j<=$3; j+=$2))
        do
                f.m "$j"
        done
}

N_PARALLEL=2
TASKS=4

main.m

for ((i=1; i<=$N_PARALLEL; i++))
do
        run_f "$i" "$N_PARALLEL" "$TASKS" &
done

wait

As a slightly more complex example, suppose N_PARALLEL is 3 and TASKS is 8. The main loop of the script will run

run_f "1" "3" "8" &
run_f "2" "3" "8" &
run_f "3" "3" "8" &

and

run_f "1" "3" "8" will run f.m 1, f.m 4 and f.m 7 sequentially.
run_f "2" "3" "8" will run f.m 2, f.m 5 and f.m 8 sequentially.
run_f "3" "3" "8" will run f.m 3 and f.m 6 sequentially.

score -1 · Answer 7 · answered Dec 11 '18 at 14:09

Citing my own answer from another question.

Here is my solution for parallel jobs which process up to N jobs at a time as configured by _jobs_set_max_parallel without the need for wait -n:

_lib_jobs.sh:

function _jobs_get_count_e {
   jobs -r | wc -l | tr -d " "
}

function _jobs_set_max_parallel {
   g_jobs_max_jobs=$1
}

function _jobs_get_max_parallel_e {
   [[ $g_jobs_max_jobs ]] && {
      echo $g_jobs_max_jobs

      echo 0
   }

   echo 1
}

function _jobs_is_parallel_available_r() {
   (( $(_jobs_get_count_e) < $g_jobs_max_jobs )) &&
      return 0

   return 1
}

function _jobs_wait_parallel() {
   # Sleep between available jobs
   while true; do
      _jobs_is_parallel_available_r &&
         break

      sleep 0.1s
   done
}

function _jobs_wait() {
   wait
}

Example usage:

#!/bin/bash

source "_lib_jobs.sh"

_jobs_set_max_parallel 3

# Run 10 jobs in parallel with varying amounts of work
for a in {1..10}; do
   _jobs_wait_parallel

   # Sleep between 1-2 seconds to simulate busy work
   sleep_delay=$(echo "scale=1; $(shuf -i 10-20 -n 1)/10" | bc -l)

   ( ### ASYNC
   echo $a
   sleep ${sleep_delay}s
   ) &
done

# Visualize jobs
while true; do
   n_jobs=$(_jobs_get_count_e)

   [[ $n_jobs = 0 ]] &&
      break

   sleep 0.1s
done

FYI, you can generate a random integer between 10 and 20 in bash with $((RANDOM%11 + 10)) (without needing to run a program).   You can get a random number between 1.0 and 2.0 with s="$((RANDOM%11 + 10))"; sleep_delay="${s%?}.${s: -1}". — G-Man Says 'Reinstate Monica', Dec 15 '18 at 22:21

Alternative to wait -n (because server has old version of bash)

7 Answers7