13

I'm running the below command, and monitoring the output file on the other system:

ssh $ip_address 'for n in 1 2 3 4 5; do sleep 10; echo $n >>/tmp/count; done'

If I kill the ssh command using ^C or by just killing the terminal I'm logged in on, I'd expect the remote command to terminate, too. This doesn't happen, though: /tmp/count gets all the numbers 1–5 regardless, and ps -ejH shows the shell and its sleep child continues running.

Is this expected behaviour, and is it documented anywhere? Can I disable it? From reading around, I'd expected to have to explicitly enable this sort of behaviour with nohup, not for it to be the default.

I've taken a look through the man pages for ssh and sshd, but not spotted anything obvious, and Google points me at instructions for turning this behaviour on, not for turning it off.

I'm running Red Hat Enterprise Linux 6.2, with a root login and bash shell on both systems.

me_and
  • 1,151
  • 1
  • 13
  • 24

7 Answers7

14

uther's answer tells you to allocate a terminal but doesn't explain why. The reason is not specific to ssh, it is a matter of signal generation and propagation. I invite you to read What causes various signals to be sent? for more background.

On the remote host, there are two relevant processes:

  • an instance of the ssh daemon (sshd), which is relaying the input and output of the remote program to the local terminal;
  • a shell, which is running that for loop.

The shell can die either naturally when it reaches the end of the loop or experiences a fatal error, or to a signal. The question is, why would the shell receive a signal?

If the remote shell is connected to sshd over pipes, which is what happens when you specify a command on the ssh command line, then it will die of a SIGPIPE if sshd exits and the shell tries to write to the pipe. As long as the shell isn't writing to the pipe, it won't receive a SIGPIPE. Here, the shell is never writing anything to its standard output, so it can live forever.

You can pass the -t option to ssh, to tell it to emulate a terminal on the remote side and run the specified command in this terminal. Then, if the SSH client disappears, sshd closes the connection and exits, destroying the terminal. When the terminal device goes away, any process running in it receives a SIGHUP. So if you kill the SSH client (with kill, or by closing the terminal that the client is running in), the remote shell is SIGHUPped.

If you pass the -t option, SSH also relays SIGINT. If you press Ctrl+C, then the remote shell receives a SIGINT.

  • 1
    Use -tt instead of -t if ssh itself does not have a tty allocated. SSH will not get an allocated tty if ssh gets backgrounded immediately upon invocation via options like -f or -n. – Miron Veryanskiy May 23 '19 at 20:00
3

Try allocating a psuedo-tty with your ssh command.

ssh -t $ip_address 'for n in 1 2 3 4 5; do sleep 10; echo $n >>/tmp/count; done'

When you disconnect the ssh session, the process should terminate.

Since this is a pseudo-tty, your shell initialization is probably not going to source configuration files, leaving your command with a very bare environment. You may need to set up a .ssh/environment file that defines environment variables such as PATH. From man 1 ssh

Additionally, ssh reads ~/.ssh/environment, and adds lines of the format 
“VARNAME=value” to the environment if the file exists and users are allowed
to change their environment.  For more information, see the 
PermitUserEnvironment option in sshd_config(5).
George M
  • 13,959
2

As an alternative to using the -t option to ssh to make the remote command terminate when the ssh client exits or (gets killed), a named pipe can be used to convert "EOF to SIGHUP" when the stdin of sshd is being closed (see Bug 396 - sshd orphans processes when no pty allocated).

# sample code in Bash
# press ctrl-d for EOF
ssh localhost '
TUBE=/tmp/myfifo.fifo
rm -f "$TUBE"
mkfifo "$TUBE"
#exec 3<>"$TUBE"

<"$TUBE" sleep 100 &  appPID=$!

# cf. "OpenSSH and non-blocking mode", 
# http://lists.mindrot.org/pipermail/openssh-unix-dev/2005-July/023090.html
#cat >"$TUBE"
#socat -u STDIN "PIPE:$TUBE"
dd of="$TUBE" bs=1 2>/dev/null
#while IFS="" read -r -n 1 char; do printf '%s' "$char"; done > "$TUBE"

#kill -HUP -$appPID 
kill $appPID

rm -f "$TUBE"
'
teru
  • 21
1

This is the best way I have found to do this. You want something on the server side that attempts to read stdin and then kills the process group when that fails, but you also want a stdin on the client side that blocks until the server side process is done and will not leave lingering processes like <(sleep infinity) might.

ssh localhost "sleep 99 < <(cat; kill -INT 0)" <&1

It doesn't actually seem to redirect stdout anywhere but it does function as a blocking input and avoids capturing keystrokes.

Relevant openssh bug: https://bugzilla.mindrot.org/show_bug.cgi?id=396#c14

0

My answer is based on teru's. I need a helper script ~/helper at the server server.ip:

#!/bin/bash
TUBE=/tmp/sshCoLab_myfifo.$$;
mkfifo "$TUBE"
( <"$TUBE" "$@" ; rm "$TUBE" ; kill -TERM 0 ) &  
cat >"$TUBE" ;
rm "$TUBE" ;
kill -TERM 0 ;

If it is called e.g. with

ssh server.ip ~/helper echo hello from server

and executes echo hello from server at server.ip, and then ssh client will terminate.

If it is called e.g. with

ssh server.ip ~/helper sleep 1000 &
CHID=$!

kill -9 $CHID will stop the script at the server also. For me, kill -INT $CHID does not work, but I do not know why.

In teru's answer, the ssh command will wait forever when the remote command is finished, because cat never finishes.

All the answers with ssh -t did not work for me with kill, but only with Ctrl-C.

Edit: I found out this worked from a new Ubuntu 14.01 to an elder scientific linux box - but not the other way round. Thus, I think there is no general solution. Weird.

0

If you want to disable that (seemingly default-behaviour) you need to enable ssh-keep-alive on the client or server-side.

If you look at the ssh-keep-alive-options in the man-pages, you can see that they are disabled by default.

Nils
  • 18,492
0

Many of the answers I've seen seem to suggest either forcing tty allocation or waiting for stdin to close. Here's a one-line solution which doesn't do either.

ssh localhost "( tail --pid=$PPID -f /dev/null; kill -HUP 0; ) >/dev/null &

The actual command

sleep 1000"

Instead of forcing a terminal, run a subshell in the background, which waits until the current sshd process ($PPID) no longer exists and sends a signal to all processes in the current process group (kill 0 in bash). Without job control, I believe all commands run in the same process group.

There may be some unnecessary overhead to this, though -- it seems tail --pid polls continuously.

Edit: The subshell also needs to not hold stdout open, in order for the ssh to exit.


Also, for anyone stumbling upon this post, here are the related questions that I've found:

Subhav
  • 21