28

This is a question that has been addressed several times, not only here, but also in other sites of the stack exchange network (e.g. How to make ssh to kill remote process when I interrupt ssh itself? ). However, I cannot make any of the solutions work for me.

I'm running a command through ssh. Whenever I exit ssh, I want the command to die as well. This command is a daemon called ktserver that runs indefinitely until you press Ctrl-C.

I run it as follows: ssh -t compute-0-1 ktserver and, indeed, when I press Ctrl-C, the process ends gracefully and the ssh session ends.

However, if instead of pressing Ctrl-C, I kill the ssh process using the kill command (for example, sending SIGINT or SIGHUP), the ktserver process stays alive.

How can I make the ktserver always die independent on how ssh is killed?

EDIT: If, instead of ktserver I run something completely different, such as gedit, everything works like a charm (i.e. gedit dies when the connection dies). Therefore, there might be something wrong with the process itself. For example, I thought that it might be ignoring SIGHUP or SIGINT. However, when I run kill -1 ktserver or kill -2 ktserver, the process dies as expected.

EDIT2: As Mark Plotnick points out, the issue is related to the fact that there is no communication circulating on the ssh channel. I've confirmed by running ssh -t <host> read and killing the ssh process afterwards. readwas still alive and kicking.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
GermanK
  • 383
  • Have you tried kill -9 ktserver? –  Dec 04 '13 at 16:12
  • @HermanTorjussen Sure, that works. The problem is that this command is run from within another process and I might not have control over all the many possibilities that might cause my process to die, and therefore the ssh session with it. So I need some reliable way to be sure that whenever my process -and therefore ssh- dies, ktserver will die with them. – GermanK Dec 04 '13 at 16:16
  • My experience with Linux is that if the remote command doesn't do any i/o to the dead tcp connection, it will keep running. I've had ssh example.com dd ... jobs run to completion even hours after the ssh connection dies due to network issues. If you can alter ktserver to take an option to output something once in awhile, that may be a workaround. – Mark Plotnick Dec 04 '13 at 17:06
  • @MarkPlotnick Indeed, I've tried running read in the remote computer, and after killing the ssh connection, read didn't die. Unfortunately, I cannot change ktserver to output anything. There is no solution then? – GermanK Dec 04 '13 at 17:23
  • Could you write a wrapper script that runs ktserver in the background, records its pid, then starts another process that periodically writes to stdout, and when the second process exits, kill the ktserver? – Mark Plotnick Dec 04 '13 at 17:35
  • Yes, I'm thinking about doing something like that. Thanks Mark! I now wonder if things like http://mosh.mit.edu/ suffer of this very same problem. – GermanK Dec 04 '13 at 18:02
  • 3
    I assumes that when ssh dies that your shell also dies. You can configure your shell to send a signal -1 (SIGHUP) when it terminates. (shopt -s huponexit). Can you test if this works for you? – Hennes Dec 05 '13 at 00:20
  • @Hennes I now realize I misspelled your nickname – GermanK Dec 16 '13 at 13:28
  • Don't use kill -9. – dimo414 Jun 19 '20 at 22:16

4 Answers4

26

I found that simply using -t -t as an argument to ssh made it work. I did not have to set huponexit to either the originating or remote shell.

I tested this as follows:

Doesn't work:

ssh user@remote sleep 100
^C

This killed the ssh session, but I can see the sleep process is still running on the remote host (ps -ef | grep sleep shows it).

Does work:

ssh -t -t user@remote sleep 100
^C

This kills the ssh session and the remote sleep process was also killed. I've also verified that the signal that is sent to the remote process is SIGINT if you use Control-C. I also verified that SIGKILL (-9) applied to the ssh process will also kill the remote process.

EDIT 1:

That was true for sleep ... for more stubborn remote processes, I found that ssh handles ^C differently that SIGINT. Ctrl-C worked, but kill -INT $pid didn'.t

Here is what I finally came up with that worked for my actual application (stealing from the other answers).

ssh -t -t -i id_rsa user@mic0 "/bin/sh -O huponexit -c 'sleep 100'"

Note the nested use of double quotes and single quotes. Note that your remote process MUST respond to SIGHUP by actually exiting!

Community
  • 1
Mark Lakata
  • 864
  • This worked best for me, but FYI, it also caused terminal control characters to bleed through, which can cause funny output. So, the easy workaround in my case was to wrap called commands and pipe their output through cat. For example, ssh -ttq user@host '{ cmd; cmd; } | cat' – Droj Jun 17 '17 at 19:22
  • Ctrl-C is NOT always equivalent to kill -INT $pid, see my answer on https://unix.stackexchange.com/questions/377191/why-are-we-using-kill-9-always/377205#377205 and another on https://stackoverflow.com/questions/8398845/what-is-the-difference-between-ctrl-c-and-sigint for the gory details ;-) – thecarpy Jul 27 '17 at 09:38
  • @thecarpy - your first answer says Ctrl-C is akin to SIGINT, and the other answer says ^C sends SIGINT. So what is the exact difference if they are not equivalent? – Mark Lakata Jul 27 '17 at 21:54
15

Usually when ssh connection dies the shell also dies. You can configure your shell to send a signal -1 (SIGHUP) when it terminates to all of its children.

For bash you can configure this option via the builtin command shopt. (shopt -s huponexit).

For zsh you want setoptHUP.

Hennes
  • 2,038
  • I thought this was the answer to my current problem, but it does not seem to work. I am running: ssh $host "scp LargeFile.dat $OtherHost:/tmp" & pid=$! and then trying to stop that transfer with: shopt -s huponexit; kill $pid , but the scp does not stop when I kill the ssh that I started. Any thoughts? – David Doria Mar 17 '14 at 17:46
  • Can you test with the shell options set before starting the scp command? Or by starting a shell, setting shopt and starting scp ? – Hennes Jun 10 '14 at 15:04
  • 3
    This works for me, but I had to first make sure I was using ssh -t -t, (notice -t twice!) which forces tty allocation, rather than just pty. – Nicolas Wu Aug 10 '14 at 12:10
1

If ssh doesn't propagate signals it receives what would you expect from it?

UPD. (special for JosephR): it's obviously an error in question itself which follows out of misunderstanding — "Kill process spawned by ssh when ssh dies". SSH doesn't spawn processes usually (sometimes it does, but this is another story), SSHD does instead, when we look at other side of connection. SSH merely relies on pseudo-terminal abstraction remote server has. That's why the only thing which can be of help there is terminal's ability to emit signals to its attached processes. This is somewhat very basic for every UNIX-like system.

poige
  • 6,231
  • 3
    This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. – Anthon Dec 17 '13 at 16:06
  • @Anthon, it's an explanation. But nobody's stating this should be the only correct answer. Thanks for your comment below. – poige Dec 17 '13 at 20:05
  • 1
    @poige Maybe flesh out the explanation so it can be of use to the OP and others. – Joseph R. Dec 18 '13 at 00:23
  • @JosephR., ok, just for you. – poige Dec 18 '13 at 01:03
  • 1
    If I say "Kill process spawned by sshd when the ssh connection is lost" would it sound better to you? I don't think that rephrasing solves my problem in any way. – GermanK Dec 18 '13 at 18:43
  • @GermanK, properly stated question contains half an answer, they say. – poige Dec 18 '13 at 18:47
  • @GermanK - issue stty -a, it should return, among other things, intr = ^C; if the interrupt key is C, I am sorry, I should have mentioned that in my comment. It can be configured to be something else. – thecarpy Jul 29 '17 at 10:04
0

The solution posted here did not work for me but since this question came up first when I was searching for solution to similar problem and also -t -t trick was mentioned here I will post solution that worked for me for others to try.

ssh -t -t -o ControlMaster=auto -o ControlPath='~/test.ssh' your_remote_ip_goes_here "your_long_running_command" &
sleep 100
ssh -o ControlPath='~/test.ssh' -O exit your_remote_ip_goes_here

My long running command was not running anymore when connection was terminated like this.

Greg0ry
  • 121
  • 6