Automatically run commands over SSH on many servers

Question

There is a list of IP addresses in a .txt file, ex.:

1.1.1.1
2.2.2.2
3.3.3.3

Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is in the known_hosts list (on my PC, Ubuntu 10.04 LTS/bash).

How can I run commands on these servers, and collect the output?

Ideally, I'd like to run the commands in parallel on all the servers.

I'll be using public key authentication on all the servers.

Here are some potential pitfalls:

The ssh prompts me to put the given servers ssh key to my known_hosts file.
The given commands might return a nonzero exit code, indicating that the output is potentially invalid. I need to recognize that.
A connection might fail to be established to a given server, for example because of a network error.
There should be a timeout, in case the command runs for longer than expected or the server goes down while running the command.

The servers are AIX/ksh (but I think that doesn't really matter.

possible duplicate of How can I reproduce commands run on one machine on another machine? — Caleb, Aug 19 '11 at 12:51
I thinks it's not a duplicate, because the link you mention doesn't even contain SSH. — LanceBaynes, Aug 19 '11 at 13:06
If you have not done so, you should set up ssh servers on all of the machines, make a private/public key pair on the machine you work from and copy the public key to the accounts on the server to prevent further password hassles. That applies for my answer, and for @demure's as well. — Anthon, May 23 '13 at 04:49

score 76 · Answer 1 · edited Dec 14 '22 at 22:38

76

There are several tools out there that allow you to log in to and execute series of commands on multiple machines at the same time. Here are a couple:

edited Dec 14 '22 at 22:38

redseven

592

answered Aug 19 '11 at 13:10

Caleb

70,105

2

I found clusterssh fairly intuitive to use. Just my 2 cents... – josinalvo Apr 13 '15 at 22:50
1

I use pssh frequently. It works very well, although I do wish I could tell it that certain remote commands will have a non-zero exit code, and that does not mean an error. That's purely cosmetic though. – acjca2 Jul 30 '17 at 14:50
1

@AnthonyClark you could append something like "|| true" to those commands. – exic Jul 02 '18 at 12:43

score 22 · Answer 2 · edited Jun 28 '16 at 15:36

If you're into Python scripting more than bash scripting, then Fabric might be the tool for you.

From the Fabric home page:

Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.

Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool.

Fabric v1 was amazing. Unfortunately, Fabric v2 removed most of the features that made it useful for this use case -- would not recommend currently (June 2018). — Meekohi, Jun 28 '18 at 15:42

score 18 · Accepted Answer · edited Oct 17 '15 at 02:34

18

Assuming that you are not able to get pssh or others installed, you could do something similar to:

tmpdir=${TMPDIR:-/tmp}/pssh.$$
mkdir -p $tmpdir
count=0
while IFS= read -r userhost; do
    ssh -n -o BatchMode=yes ${userhost} 'uname -a' > ${tmpdir}/${userhost} 2>&1 &
    count=`expr $count + 1`
done < userhost.lst
while [ $count -gt 0 ]; do
    wait $pids
    count=`expr $count - 1`
done
echo "Output for hosts are in $tmpdir"

edited Oct 17 '15 at 02:34

iruvar

16,725

answered Aug 19 '11 at 14:55

Arcege

22,536

1

what does exactly wait do in this script?? ty! – LanceBaynes Aug 19 '11 at 15:43
what happens if a servers key is not in my knows_hosts file? – LanceBaynes Aug 19 '11 at 15:50
5

putting scripts into the background creates child processes. When a child process exits, the process 'slot' and resources stay in the system until the parent process exits or the parent process 'waits' for the child. These 'terminated by still present' processes are called 'zombie' processes. It is good behavior to clean up after child proceses and reclaim resources. – Arcege Aug 19 '11 at 19:38
1

If not known, then that might screw it up, but you can add -o StrictHostKeyChecking=no to avoid this. However it is better to scan all the servers from a command-line first so the host keys can be added. – Arcege Aug 19 '11 at 19:40
sry, for asking this again, but what does this part: http://pastebin.com/raw.php?i=iYPYgyHg do exactly?? :O – LanceBaynes Aug 28 '11 at 17:48
1

As I mentioned, after the ssh program is put into the background and after it exits, resources are still kept. The wait command reclaims those resources, including the process' return code (how the processs exited). Then if later in the program, you put another process, lets say a compression, in the background and needs to wait for it, then without this loop, the wait will reclaim one of the completed ssh programs, not the compression - which may still be running. You would get an return status on the wrong child process. It is good to clean up after yourself. – Arcege Aug 28 '11 at 18:23
@arcege Zombies do not hold resources beyond an entry in the process table. – Graham Nicholls Oct 22 '21 at 11:31

score 15 · Answer 4 · edited Aug 06 '20 at 07:58

15

I do use GNU parallel for that, most specifically you can use this recipe:

parallel --tag --nonall --slf your.txt command

With your.txt being the file with the server IP address/names.

edited Aug 06 '20 at 07:58

Mathias Brodala

105

answered May 23 '13 at 04:45

Anthon

79,293

isn't there any other way only using ssh Since I am using my company servers I don't want to install additional packages – Özzesh May 23 '13 at 04:49
Sure, I used to do this before ssh existed as well, but you need some way to login to the other computers and those ways used to be less secure (like rsh). You don't describe what you are using now to get from machine to machine (telnet?, rsh?). – Anthon May 23 '13 at 04:52
I am using ssh connection to all servers – Özzesh May 23 '13 at 05:03
Sorry I misunderstood, I read it as if you did not have ssh installed on the servers and did not want to install it. AFAIK you need parallel only on the controlling machine, it uses ssh to get to the others. But I am not sure, I have it on all my machines. – Anthon May 23 '13 at 05:05
so only 1 machine with gnu_parallel will work? – Özzesh May 23 '13 at 05:16
1

@Özzesh yes only on the controller – Anthon May 23 '13 at 05:20
2

@Özzesh See if your reason for not installing GNU Parallel is covered on http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html – Ole Tange May 23 '13 at 08:07
@OleTange --slf is definately better than -S $(cat your.txt), I use this most often from Makefiles and don't have that many other machines to put the list in a file. I will update my answer. Thanks – Anthon May 23 '13 at 08:12
1

@OleTange Just realised who commented on my answer %-). Thanks for this excellent piece of software. – Anthon May 23 '13 at 08:18
seems not support run sudo behind ssh – yurenchen Sep 23 '22 at 01:02

michas · Answer 5 · 2013-05-27T11:19:51.737

14

Very basic setup:

for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done

Authenticating with name/password is really no good idea. You should set up a private key for this:

ssh-keygen && for host in $(cat hosts.txt); do ssh-copy-id $host; done

edited May 27 '13 at 11:19

answered May 23 '13 at 07:00

michas

21,510

1

Thanks - I was looking for a 1-liner. – thebiggestlebowski Oct 28 '20 at 17:00

score 4 · Answer 6 · edited May 27 '13 at 18:19

4

I suggest Ansible.cc. It's a configuration manager and command dispatcher.

edited May 27 '13 at 18:19

Michael Mrozek

93,103
40
240
233

answered May 27 '13 at 11:33

Tom

57

Andrew Childs · Answer 7 · 2014-05-23T02:13:41.640

2

I built an open-source tool called Overcast to make this sort of thing easier.

First you define your servers:

overcast import vm.01 --ip 1.1.1.1 --ssh-key /path/to/key
overcast import vm.02 --ip 2.2.2.2 --ssh-key /path/to/key

Then you can run multiple commands and script files across them, either sequentially or in parallel, like so:

overcast run vm.* uptime "free -m" /path/to/script --parallel

edited May 23 '14 at 02:13

answered May 10 '14 at 20:56

Andrew Childs

141

score 2 · Answer 8 · answered Dec 04 '14 at 21:13

The Hypertable project has recently added a multi-host ssh tool. This tool is built with libssh and establishes connections and issues commands asynchronously and in parallel for maximum parallelism. See Multi-Host SSH Tool for complete documentation. To run a command on a set of hosts, you would run it as follows:

$ ht ssh 1.1.1.1,2.2.2.2,3.3.3.3 uptime

You can also specify a host name or IP pattern, for example:

$ ht ssh 1.1.1.[1-99] uptime
$ ht ssh host[00-99] uptime

It also supports a --random-start-delay <millis> option that will delay the start of the command on each host by a random time interval between 0 and <millis> milliseconds. This option can be used to avoid thundering herd problems when the command being run accesses a central resource.

score 2 · Answer 9 · answered Aug 19 '11 at 13:08

2

I think you're looking for pssh and the related parallel versions of the usual scp, rsync, etc..

answered Aug 19 '11 at 13:08

unclejamil

914

score 1 · Answer 10 · answered Jan 18 '16 at 01:59

Just a headsup for a really nice question:

Best solution I've found is, not-so-surprisingly, tmux.

You can do Ctrl-B : setw synchronize-panes in tmux for amazing resut. You must have all of your ssh connections opened, in different panes of 1 window of Tmux for this.

score 1 · Answer 11 · answered Oct 25 '21 at 18:46

1

Use pdsh. Here's a useful example:

pdsh -w 'something.example.com,node-[1-5].example.com' -R ssh 'uptime'

answered Oct 25 '21 at 18:46

Nowaker

162

seems not easy to input password for sudo run (hava to call sudo --stdin by user) – yurenchen Sep 23 '22 at 00:55

Y0NG · Answer 12 · 2018-02-13T08:04:37.900

0

Maybe something like this works, to run command on multiple hosts? suppose all hosts are setup in .ssh/config for password-less login.

$ < hosts.txt xargs -I {} ssh {} command

edited Feb 13 '18 at 08:04

answered Feb 13 '18 at 07:58

Y0NG

9

score 0 · Answer 13 · answered Mar 20 '18 at 22:31

0

Create a file /etc/sxx/hosts

populate like so:

[grp_ips]
1.1.1.1
2.2.2.2
3.3.3.3

share ssh key on all machines.

Install sxx from package:

https://github.com/ericcurtin/sxx/releases

Then run command like so:

sxx username@grp_ips "whatever bash command"

answered Mar 20 '18 at 22:31

ericcurtin

241

Where is the output written? How are non-zero exit statuses handled? Please do not respond in comments; [edit] your answer to make it clearer and more complete. – G-Man Says 'Reinstate Monica' Mar 20 '18 at 23:38

score 0 · Answer 14 · edited Oct 24 '18 at 14:10

Use Paramiko http://www.paramiko.org/ and Thread-based parallelism https://docs.python.org/3/library/threading.html .below code allows to execute multiple commands on each server in parallel!

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.load_system_host_keys()
ssh.connect(hostname, port, username, password)
t = threading.Thread(target=tasks[index], args=(ssh, box_ip,))
t.start()

Note: repeat above instructions for each server.

score 0 · Answer 15 · answered Aug 01 '19 at 07:07

0

I think "expect" is what you need.
Many people does not even know it exists. It's a tcl based program that will made question and possible answers on your behalf. Have a look to https://wiki.tcl-lang.org/page/Expect

answered Aug 01 '19 at 07:07

Clement

57

1

Also, you could use the autoexpect script to record one sample interaction and then edit the resulting file to adjust as needed. – Fjor Apr 28 '22 at 16:01

redseven · Answer 16 · 2022-12-13T21:47:36.377

I would suggest to use Ansible ad hoc commands for this job. Ansible uses ssh and anyway it is a nice tool to manage multiple hosts, but now I am just focusing on the topic to run a single command on multiple hosts.

Requirement

your user has an ssh key already installed on the remote hosts (and for most command you want passordless sudo rights there as well). So the following command should work without asking any password on any of your managed hosts (running uptime command on the remote server):

ssh server1.mydomain.com uptime

or even better (to check passwordless sudo):

ssh server1.mydomain.com sudo uptime

Configure ansible

As the ssh works fine you can create an inventory file for ansible: inventory.yml (yes, it's in YAML format, so whitespace matters):

all:
  hosts:
    server1.mydomain.com:
    server2.mydomain.com:
    server3.mydomain.com:

Run command on multiple hosts

and now run ansible to use uptime command on all of your hosts in the inventory:

ansible -i inventory.yml --one-line all -a uptime

output:

server1.mydomain.com | CHANGED | rc=0 | (stdout)  14:48:24 up 5 days,  9:18,  1 user,  load average: 0.16, 0.04, 0.01
server2.mydomain.com | CHANGED | rc=0 | (stdout)  12:48:24 up 6 days,  7:17,  3 users,  load average: 0.02, 0.12, 0.17
server3.mydomain.com | CHANGED | rc=0 | (stdout)  14:48:24 up 5 days,  9:18,  1 user,  load average: 0.08, 0.02, 0.01

The --one-line option is not a must, just gives you nicer output if you have many servers and the output really just one line.

If the user on the remote server is not the same as your local user you can define that in the inventory or in the command line as well:

ansible -i inventory.yml -u remoteuser all -a uptime

If you need arguments for the command use quotation marks:

ansible -i inventory.yml all -a "ls -lh /etc"

And if you have to use pipe (|) than you need the shell module:

ansible -i inventory.yml all -m shell -a "cat /etc/passwd | grep root"

And finally for the original question, use this command before ansible to accept any ssh host keys:

export ANSIBLE_HOST_KEY_CHECKING=False

No, ansible doesn't need python 2 any more: https://docs.ansible.com/ansible/latest/reference_appendices/python_3_support.html — redseven, Dec 13 '22 at 13:27

Automatically run commands over SSH on many servers

16 Answers16

Requirement

Configure ansible

Run command on multiple hosts

Linked

Related