1

I have 1000s of remote machines. I want to run some specific commands on all of them, but in parallel. I use these commands which run sequentially:

for f in `cat host.lst`
do
       ./runScript.sh $f
done

Let's suppose host.lst contains 100 hosts. I want to run runScript.sh on 100 hosts in parallel. Also, logs should be maintained.

I can not install any utility on my machine such as PSSH.

I have done a lot of research and found these links but they did not help. I do not understand how they work:

Automatically run commands over SSH on many servers

Execute command on multiple files matching a pattern in parallel

Can any one explain the logic?

Community
  • 1
api1411
  • 124
  • 1
    There are answers in both the questions that you referenced that don't require pssh. Some use GNU parallel (which would be my choice) but there are others that use only commands that are commonly installed on Unix-like systems. I'd advise that you [edit] the question to show which of those answers you've tried and detail the problems you encountered. As it stands, this question is a duplicate. – Anthony Geoghegan Apr 12 '17 at 08:55
  • See if you understand https://www.gnu.org/software/parallel/parallel_tutorial.html#Remote-execution You do not need to be root to use GNU Parallel - even if it is not installed: http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html – Ole Tange May 04 '17 at 22:35

2 Answers2

6
logdir=`mktemp -d`
bunch=200

IFS=$'\n'
for hosts in $(< hosts.lst xargs -r -L "$bunch"); do
   IFS=" ";
   for host in $hosts; do
      ssh -n -o BatchMode=yes "$host" './runScript.sh' 1>"$logdir/$host.log" 2>&1 &
   done
   wait
done

Assuming the 1000s (thousands) of hosts are listed one/line in the hosts.lst file and then from these a bunch are selected in one time (200), and on each of these 200 hosts are spawned your runScript.sh using ssh in batch mode and at the same time preserving the stdout+stderr spewing forth from each of these backgrounded job into a file with the name of host in the directory $logdir, which may be examined as and when required.

Finally we wait for one bunch to get over before we launch the next bunch, by means of the wait command at the end of the inner for loop.

Coddy
  • 113
  • Rakesh, i think you were trying to write 1000s ???? u wrote 100s. – api1411 Apr 12 '17 at 22:18
  • I just want to trigger the runscript only. so can " ssh -n -o Batch mode=yes "$host" " can be removed?? – api1411 Apr 12 '17 at 22:21
  • what do you mean trigger runscript.sh? If you want to remve ssh then why bother about that hosts list ? –  Apr 12 '17 at 22:44
  • sorry "$host" is needed runscript.sh will take care of every thing like scp, rsync and all. I just need to trigger runscript.sh. – api1411 Apr 13 '17 at 00:51
  • Thanks Rakesh, It worked !! last question, Can i use exec in place of xargs?? What would be the impact.. – api1411 Apr 13 '17 at 04:31
  • I don't think exec is the appropriate tool in this scenario as exec replaces it's parent by itself. xargs can be useful if your version supports the -P flag. –  Apr 13 '17 at 05:14
2

The following modification to your script, which doesn't require any other utilities, will execute in parallel, but may run into limits depending on your hosts' limitations of open file handles:

for f in $(cat host.lst); do
       ./runScript.sh $f &
done

To capture the results to a log, just add > run.log after done to save to a new file, run.log.

The key change is the addition of &, which runs a process in the background rather than waiting for it to complete before executing the next command.

DopeGhoti
  • 76,081