How to skip unresponsive server

Question

I wrote a script that runs on multiple servers. Sometimes the script gets hung on one of the servers and I have to hit (control C) to end the process. If not, it gets stuck and keeps trying to connect.

If/when a server gets hung or unresponsive when running a script, is there a way to skip that host so the script can go to the next host and keep running along? Usually When I hit control C, that ends the entire process.

Here's an example of the script. Let's say it gets hung on machine 3.

HOSTS=(MACHINE1 MACHINE2 MACHINE3 MACHINE4 MACHINE5)
for i in "${HOSTS[@]}"
do
  echo "$i"
  ssh -q "$i" uname -a
 done

This script is being run on OS X. I tried using the timeout command but unfortunately, it does not work.

Can you explain how you run the script on each server? e.g. the script SSH's to each server and runs command, or you log onto each server and run it manually? — Banjer, Aug 27 '15 at 17:49
If it's ssh you should be able to set a timeout or use batch mode to help. — Centimane, Aug 27 '15 at 18:33
There is actually a timeout commmand, e.g. timeout 20 ssh blah — doneal24, Aug 27 '15 at 18:37
@Banjer, here's an example of the script..and lets say it gets hung on machine 3 — Sad Puppy, Aug 27 '15 at 20:24
HOSTS=(MACHINE1 MACHINE2 MACHINE3 MACHINE4 MACHINE5)
for i in "${HOSTS[@]}" do echo "$i" ssh -q "$i" uname -a done — Sad Puppy, Aug 27 '15 at 20:25
This script is being run on OS X. I tried using the timeout command but unfortunately, it does not work — Sad Puppy, Aug 27 '15 at 21:46

score 2 · Answer 1 · edited Apr 13 '17 at 12:36

Rather than roll your own and have to cope with everything that can go wrong (host not responding, host stopping responding in the middle, user pressing Ctrl+C, error reporting, …), use one of the many existing tools to run a command on many machines over SSH.

mussh -t 4 -H <(printf '%s\n' "${HOSTS[@]}") -c 'uname -a'
pssh -t 4 -h <(printf '%s\n' "${HOSTS[@]}") uname -a
pdsh -u 4 -w "$(printf %s, "${HOSTS[@]}")" 'uname -a'
…

Mark Plotnick · Accepted Answer · 2015-08-28T08:00:34.517

1

A typical way to do this is to use the trap command to tell the shell script to ignore SIGINT (generated by Control-C), and then to re-enable SIGINT in a subshell just before your command is run.

trap "" INT
HOSTS=(MACHINE1 MACHINE2 MACHINE3 MACHINE4 MACHINE5)
for i in "${HOSTS[@]}"
do
    echo "$i"
    (trap - INT; ssh -q "$i" "uname -a")
done

edited Aug 28 '15 at 08:00

answered Aug 27 '15 at 17:58

Mark Plotnick

25,413
3
64
82

How to skip unresponsive server

2 Answers2