0

I have a usecase where i have to excute some commands which are long running, so i have to run 3-6 commands in parallel so that i will finish in as fast as possible.

Below is my code snippet:

#!/bin/sh
executeCommand(){
  //command which will run for almost 2-3 min
}

executeCommand 1 &
P1=$!
executeCommand 2 &
P2=$!
executeCommand 3 &
P3=$!
wait $P1 $P2 $P3

What i observed when running this block, P3 runs first and then after 30 sec, P2 and P3 ran at same time. I actually need to start all at once only so that it finish as fast as possible.

Below is my machine configuraions: EC2 instance : p3.2xlarge(8vCPU's, 61GB RAM,1 Nvidia Tesla V100 GPUs, 16 GPU Memory (GB))

Can someone suggest what is best to run all command at once?

Thanks in advance

Sumit Bisht
  • 1
  • 2
    Could you give more information about "some command which will run for almost 2-3 min"? Do you get the same behaviour if you replace it with something simpler (like timeout 10 yes "$1")? – A.B Mar 18 '21 at 15:47
  • Those commands are video extractions from several different bitrates. Command will take around 2-3 min. So i have to run for several different files so i was thinking to run all in parallel, so that it can finish as fast as possible. – Sumit Bisht Mar 19 '21 at 05:08
  • I have to ask the same: do you get the same behaviour if you replace it with something simpler (like timeout 10 yes "$1")? – A.B Mar 19 '21 at 08:43
  • 2
    Do the "long running commands" compete for resources, e.g. exclusive access to a file? – waltinator Mar 19 '21 at 16:45
  • Replace wait ... with while : ;do ps -fp$P1,$P2,$P3 ;sleep 1;done to debug. Read man ps. – waltinator Mar 19 '21 at 17:02
  • 1
    I'm closing this as "unclear" since the observed behavior could depend on the actual actions taken by the shell function, which we know nothing about. – Kusalananda Mar 22 '21 at 10:37

1 Answers1

2

Your code is correct.

I think you are mistaken when you say P2 and P3 "wait" for P1 to finish before starting.

Here is a proof (and you can actually test the same thing by adding the same debug messages within your actual code to be sure):

executeCommand(){
  echo "Starting $1"
  sleep 4
  echo "$1 Complete"
}

executeCommand 1 &
p1=$!
executeCommand 2 &
p2=$!
executeCommand 3 &
p3=$!
wait "$p1" "$p2" "$p3"

$ sh yourScript
Starting 1
Starting 2
Starting 3
1 Complete
2 Complete
3 Complete
$
Camusensei
  • 223
  • 2
    P2 and P3 cold appear to wait for P1 to be done depending on what these processes do. If P1 uses massive amounts of disk I/O bandwidth, then P2 and P3 could be delayed. If there is file locking involved, where P1 get an exclusive lock for the duration of its run, P2 and P3 would also have to wait. – Kusalananda Mar 21 '21 at 20:31
  • In that case the problem would be way different from a shell-script question. Let's wait for OP to confirm or not the problem – Camusensei Mar 22 '21 at 10:25
  • 1
    Or P1 could use a GPU to do video encoding acceleration: P2 or P3 might be locked out from using this resource meanwhile etc. – A.B Mar 22 '21 at 10:42
  • Got the answer, Actually it was problem with SMB version. We were using SMB ver 2.0 when upgraded to SMB version 2.1 it got resovled. – Sumit Bisht Jul 21 '21 at 09:38
  • You can mark my answer as solved then if it's correct and helped out ;) – Camusensei Jul 22 '21 at 10:09