HPC ssh "connection closed by remote host"

Question

My HPC installed LSF job scheduler.

I logon the login node (I use xshell) and using interactive job submission command

bsub -Is csh

Thus, I entered one of the HPC node, for example c01 node

Then I want to enter another node, for example c02, so I use

ssh c02

I successfully entered c02 node. But after several minutes, the connection is closed. The message is

Connection to c02 closed by remote host. 
Connection to c02 closed.

So how to maintain this connection?

The following message is generated when using ssh -vvv c02

debug3: Wrote 64 bytes for a total of 2925
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i0/0 o0/0 fd 4/5 cfd -1)

debug3: channel 0: close_fds r 4 w 5 e 6 c -1
Connection to c02 closed by remote host.
Connection to c02 closed.
Transferred: sent 2744, received 2384 bytes, in 158.3 seconds
Bytes per second: sent 17.3, received 15.1
debug1: Exit status -1

There is insufficient information to answer the question. In all likelihood, you're hitting some sort of timeout. Try looking at the sshd logs on c02. — Jesse K, Oct 19 '15 at 14:43
@JesseKeilson Thank you for comment. How to see the sshd logs on c02? — user15964, Oct 19 '15 at 14:45
Depends on your sshd configuration. Probably /var/log or /var/log/secure, but could be anywhere. — Jesse K, Oct 19 '15 at 14:48
This might also be a network device in between deciding that an idle connection should be closed. — Thorbjørn Ravn Andersen, Nov 26 '20 at 11:57

ss333 · Answer 1 · 2015-10-19T16:37:20.453

0

You can try SSH keep alive. On c01 create or add the following to ~/.ssh/config

host c02
  ServerAliveCountMax 30
  ServerAliveInterval 60
  TCPKeepAlive yes

This will keep send a TCP keepalive message every 60 seconds for 30 minutes. https://www.freebsd.org/cgi/man.cgi?query=ssh_config&sektion=5

Make sure to set the permissions as below:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/config

These options can also be put on the command line, but I find a config file makes things easier. If you want to apply them to every host then instead of host c02, use host *. Since there is probably a shared file system, this will work on all your ssh connections.

Note: These options could be disabled in sshd_config, if it is then they won't work.

Another option that might work is run an interactive command like: watch ls ~

edited Oct 19 '15 at 16:37

answered Oct 19 '15 at 15:48

ss333

21

I tried to edit ~/.ssh/config. Not working. where is sshd_config? I look into /etc/ssh/ssh_config, I don't know which option cause the disconnection. – user15964 Oct 19 '15 at 16:31
You will have to be an administrator on the machine to edit sshd_config. It is typically in /etc/ssh/sshd_config. You may or may not have permission to view that file. I added a note about permissions to my answer, ssh is very picky, that might help. – ss333 Oct 19 '15 at 16:35
Thank you. But I found the sshd_config is empty. So it should not be the problem of sshd_config. – user15964 Oct 20 '15 at 08:55
Are you an admin(do you have sudo/root) on c02? – ss333 Oct 22 '15 at 18:05
No, I don't have an admin. I also tried ssh -o ServerAliveInterval 60. still failed. – user15964 Oct 23 '15 at 03:52

HPC ssh "connection closed by remote host"

1 Answers1

Linked