Execute remote commands, completely detaching from the ssh connection

Question

I have 2 computers, localpc and remoteserver.

I need localpc to execute some commands on remoteserver. One of the things it needs to do is start a backup script that runs for a number of hours. I would like the command on localpc to “fire” and then be running totally independent on remoteserver, like localpc was never there in the first place.

This is what I have done so far:

remoteserver contains has the script:

/root/backup.sh

localpc is scheduled to run this:

ssh root@remoteserver 'nohup /root/backup.sh' &

Am I doing this the right way? Is there a better way to do this? Will I run into any trouble doing it this way?

http://stackoverflow.com/questions/19996089/use-ssh-to-start-a-background-process-on-a-remote-server-and-exit-session and http://stackoverflow.com/questions/29142/getting-ssh-to-execute-a-command-in-the-background-on-target-machine provide a number of useful approaches for this problem not using screen/tmux etc... — Paul, Jul 06 '16 at 12:42

Gilles 'SO- stop being evil' · Answer 1 · 2012-02-01T10:50:14.793

64

Close, but not exactly.

Independently of any terminal

ssh root@remoteserver '/root/backup.sh </dev/null >/var/log/root-backup.log 2>&1 &'

You need to close all file descriptors that are connected to the ssh socket, because the ssh session won't close as long as some remote process has the socket open. If you aren't interested in the script's output (presumably because the script itself takes care of writing to a log file), redirect it to /dev/null (but note that this will hide errors such as not being able to start the script).

Using nohup has no useful effect here. nohup arranges for the program it runs not to receive a HUP signal if the program's controlling terminal disappears, but here there is no terminal in the first place, so nothing is going to send a SIGHUP to the process out of the blue. Also, nohup redirects standard output and standard error (but not standard input) to a file, but only if they're connected to a terminal, which, again, they aren't.

Detaching from a terminal

 aaron@localpc$ ssh root@remoteserver
 root@remoteserver# nohup /root/backup.sh </dev/null &
 nohup: appending output to `nohup.out'
 [1] 12345
 root@remoteserver# exit
 aaron@localpc$

Use nohup to detach the script from its controlling terminal so that it doesn't receive a SIGHUP when the terminal disappears. nohup also redirects the script's standard output and standard error to a file called nohup.outif they're connected to the terminal; you have to take care of standard input yourself.

Keeping a remote terminal

If you want to keep the command running in a remote terminal but not have it attached to the SSH session, run it in a terminal multiplexer such as Screen or Tmux.

ssh root@remoteserver 'screen -S backup -d -m /root/backup.sh'

You can later reconnect to the terminal where the script is running by invoking screen -S backup -rd as root on that machine.

Automating one remote command

For slightly better security, don't open direct remote root logins too widely. Create a special-purpose key pair and give it a forced command in /root/.ssh/authorized_keys. The contents of the public key file is AAAA…== wibble@example.com; add a comma-separated list of options including command="…" which specifies that the key can only be used to execute this specific command. Be sure to keep the options and the key all on one line.

command="/root/backup.sh </dev/null >/dev/null 2>/dev/null &",no-port-forwarding,no-agent-forwarding,no-x11-forwarding,no-pty,no-user-rc AAAA…== wibble@example.com

edited Feb 01 '12 at 10:50

answered Jan 31 '12 at 00:14

Gilles 'SO- stop being evil'

829,060

I have tried the first command you listed. It does not background on the box that executed the command. The cursor just sits there and waits for completion... perhaps it's a bug? – LVLAaron Jan 31 '12 at 13:21
You might need to redirect stdin to /dev/null on the remote system. That may be why ssh isn't exiting. – KeithB Jan 31 '12 at 20:20
I would add the -f option to get ssh to ‘background’ (i.e. terminate, close connections) on the local side. This will work in conjunction with the &. nohup is optional in this case, but you might consider using setsid instead. – Alexios Feb 01 '12 at 09:36
@KeithB Redirecting stdin is part of it, but I also need to redirect stdout and stderr: nohup won't do it here because they're not terminals, and in fact nohup isn't useful here. – Gilles 'SO- stop being evil' Feb 01 '12 at 10:57
@Alexios Using -f is not useful unless a terminal is needed for authentication, and I'm not going to recommend that. With ssh -f, the ssh client is detached from the terminal on localpc, but it keeps running in the background for no useful purpose. – Gilles 'SO- stop being evil' Feb 01 '12 at 10:59
@Gilles That's very true. Just hedging bets. I'm too busy to check the SSH source code, but will the remote daemon exit if the process has forked, detached from all open FDs and/or spawned a new SID? Scratch that, of course it should. For all intents and purposes, it would be done processing its ‘session’. – Alexios Feb 01 '12 at 11:14
While all else might be correct, as far as I know ssh connections build virtual terminals on both client and host. So saying that there is no terminal is incorrect, there are two. But it might still be true that nothing sends a hup – erikbstack Mar 13 '15 at 10:20
@erikb85 An ssh connection doesn't create a virtual terminal on the client side (why would it?), and creates a terminal on the sever side only with the -t option or if no command is passed. – Gilles 'SO- stop being evil' Mar 13 '15 at 11:48
@Gilles Well I'm no expert, but people who program on the tty level told me so. Additional evidence points in the same direction: a) stty -echo works b) My man page reads If a pseudo-terminal has been allocated (normal login session)[...] – erikbstack Mar 16 '15 at 09:14
@erikb85 That pseudo-terminal is on the server side, not on the client side. – Gilles 'SO- stop being evil' Mar 16 '15 at 13:07
@Gilles And that does or does not send a SIGHUP when it closes? I'd be really curious about how this works and where to get such kind of information. Such kind of low level topics often does not yield many google results, at least the way I google. – erikbstack Mar 18 '15 at 12:25
@erikb85 I don't understand your question: send SIGHUP to what? The processes on the server get a SIGHUP when the connection goes down, because the terminal goes away at that point (the sshd process that manages the server side of the connection exits when the connection is severed). There are several threads about the intricacies of SIGHUP, shell jobs and process groups on this site. – Gilles 'SO- stop being evil' Mar 18 '15 at 12:53
@Gilles Your answer that we are currently commenting on says you don't need nohup because your shell command does not receive any SIGHUPs. But now we find through discussion that sshd creates a pseudo terminal for the shell command. And if I understand you correctly closing the connection will result in the command receiving a SIGHUP from the closing pseudo terminal. That means nohup might still have a reason to be used. – erikbstack Mar 19 '15 at 11:24
1

@erikbwork Please note that I have absolutely no clue about those things, but following the discussion and the comments: Gilles does not use the -t option in his commands, so no pseudo terminal will be established, neither at the client nor at the server side. Therefore, no HUP will be sent. – Binarus May 15 '17 at 18:30

score 56 · Accepted Answer · edited Feb 01 '12 at 10:53

56

You should probably use screen on the remote host, to have a real detached command:

ssh root@remoteserver screen -d -m ./script

edited Feb 01 '12 at 10:53

Gilles 'SO- stop being evil'

829,060

answered Jan 31 '12 at 22:04

enzotib

51,661

I like this idea. When the script finishes, screen will still be running through... and I'd have to somehow reconnect to it next time I wanted to run it, right? – LVLAaron Feb 01 '12 at 10:06
@AaronJAnderson: no, given that the command is not a shell, when it terminates also that screen session terminates. – enzotib Feb 01 '12 at 10:12

score 13 · Answer 3 · edited Jul 08 '14 at 16:10

The standard recipe for running a remote command from a remote login like SSH is the following:

nohup command </dev/null >command.log 2>&1 &

If command is a shell script that takes care of logging to a file itself, then you could change command.log to /dev/null. Once you start this, log off right away.

You need everything on that line.

nohup tells the shell not to disturb the process if the login session is disconnected.

</dev/null tells it never to wait for input

>command.log tells it to send any messages to this named log file

2>&1 tells it to send any stderr messages to the same log file. In some cases it is better to have two files, the second one to collect error messages, and the first to collect normal activity messages. That can make it easier to verify that everything worked correctly.

& tells it to detach this process and run it in the background as a daemon process.

score 13 · Answer 4 · edited Apr 13 '12 at 22:53

This thread was very helpful but my solution had to be a bit different.

I don't like the screen solution because it leaves a screen process running which I don't need. The redirections and nohups used like this:

ssh root@remoteserver '/root/backup.sh </dev/null >/var/log/root-backup.log 2>&1 &'

were NOT working for me when used with the ssh command.

I created a wrapper script on the remote machine which runs the actual script. The wrapper script sets the redirection and nohup. Something like this:

backupwrapper.sh:
nohup backup.sh > /dev/null 2>&1 &

Then on my client machine I run (notice no redirection):

# ssh remotemachine "backupwrapper.sh"
#

The ssh command immediately returns, the connection is terminated and the script is left running.

score 7 · Answer 5 · edited Apr 13 '17 at 12:36

As Nils says, it's a security risk to allow root to log in via ssh. And I advise against running any substantial job on a machine without a log. If anything goes wrong, you'll wish you had some troubleshooting messages. There are other answers showing you how to do that. But here's how I recommend accomplishing just what you asked for. It's all built into sh(1). No need for GNU screen (although I think that's a clever solution). Append this string to your command: >&- 2>&- <&- &. >&- means close stdout. 2>&- means close stderr. <&- means close stdin. & means run in the background, e.g.

$ ssh myhost 'sleep 30 >&- 2>&- <&- &'
# ssh returns right away, and your sleep job is running remotely
$

score 0 · Answer 6 · edited Feb 01 '12 at 10:56

You should background the remote command on remoteserver. The way you do it will background the ssh-command on localpc.

Apart from that it is a bad idea to allow root-ssh.

So setup on remoteserver: A sudoers line that will run /root/backup.sh as root by user backup.

You could create that user without "good" password.

Put the command sudo /root/backup.sh & as command into ~backup/.ssh/authorized_keys — together with the public key from localpc (whoever triggers that script).

score -1 · Answer 7 · edited Feb 01 '12 at 10:53

-1

#screen -d -m -S BACKUP ssh root@remoteserver /root/backup.sh

edited Feb 01 '12 at 10:53

Gilles 'SO- stop being evil'

829,060

answered Feb 01 '12 at 09:06

Killinks

11

2

That's not very useful: if the SSH connection dies, the script's stdout and stderr will be closed, which may cause trouble. Screen should be running on the remote machine (as in enzotib's answer). – Gilles 'SO- stop being evil' Feb 01 '12 at 10:54

Execute remote commands, completely detaching from the ssh connection

7 Answers7

Independently of any terminal

Detaching from a terminal

Keeping a remote terminal

Automating one remote command

Linked

Related