Weird SSH issue, ssh works with -t but freezes without it

Question

When I ssh into one of my servers, it seems to log in, but then hangs before giving me the prompt (message debug2: shell request accepted on channel 0 is the last log entry).

Though the odd thing is ssh -t "/bin/bash" works when ssh doesn't.

What I have found out so far

• I can log in fine from servers in the same geographical location normally
• If I ssh -t '/bin/bash' - I can log in perfectly from ANY location.
• If I use rsync to the server , it seems to work, and then locks
• If I use rsync from the server, it works without problem

What I have tried

• removing or changing all login options .profile, .bashrc /etc/profile
• Changing the ssh_config and/or sshd_config to one from an identical server that works fine
• I've checked the routing
• I've had a network expert examine tcpdump to no avail (though there do seem to be a lot of re-transmissions)

I really can't think of anything else

Apart from a dodgy network card driver/firmware.

Answer 1

That may comes from a problem in the profile.
When you connect with ssh -t /bin/bash your shell will not 'login', it will not source the /etc/profile nor the ~/.profile and ~/.bashrc...

So after connecting, put the shell in debug mode then source each file to find what is blocking inside :

set -x 
. /etc/profile 
...and so on

EDIT

Note that I was wrong, the .bashrc file will be sourced by any interactive shell (so only the profile, profile.d files matter here).

Try to make the list of the different phases of the connection process. Something like this

1) ssh connecting (config,...)
2) login (PAM, tty, wtmp, ...)
3) starting the shell (profile, home dir access, ... )

To check the (1) you could start the sshd daemon in debug mode. For that you need to start another sshd, listening to a dedicated port (not on port 22 so no need to stop the regular sshd daemon) .

# /usr/sbin/sshd -p 2222 -ddd 

That sshd will accepts only one connection and will not goes into background. Open another terminal and connect to that ssh session.

# ssh -vvv -p 2222 user@host

You could compare the messages you get with the one of another server of the same brand. Then you will know if the problem is on the ssh side or not.

(-ddd and -vvv are the maximum debug level, you can tune that)

I got that link, much more detailed.

Answer 2

The answer to my problem Turns out it was a networking issue. I eventually found out that by dropping the MTU of all the network cards down a bit, the problem went away.

Most likely something is misconfigured for that network, but I can now prove that and hand it over to the network team (who kept telling me it was a server issue).

Be aware though, this is a last resort The peculiarity I had was that ssh server worked from same subnet but not outside it, and ssh -t '/bin/bash' worked from anywhere.

I also had rsyncs that would start fine, but at some point just hang, or drop the connection.

So if you are looking at this answer, try the others above first, and ONLY if you get oddities like mine, is this likely to help.

So thanks for all the help. I tried everything, and it taught me loads, and let me confirm it had nothing to do with the server (which is everything I had hoped for).

So thank you to everyone who helped

Answer 3

I have same problem when I used a nested virtualization platform recently.

It works after reducing MTU from 1500 to 1400 on the source or destination server.

/sbin/ip link set dev eth0 mtu 1400

Answer 4

When you do ssh some_user@some_host /bin/bash, what you are doing is launching some_user's shell (as defined in /etc/passwd on some_host) and then executing the given command, /bin/bash, from within that shell.

Now, some_user's shell (let's assume it's also bash) isn't being launched interactively (it's executing the given command instead). So it doesn't allocate a pty (a pseudo-terminal) and this means there isn't a pty for the issued command to use so it starts non-interactively.

In the example, the requested /bin/bash command is launched but it appears to hang.

You can fix this by asking ssh to create a pty anyway so that there is one available for the shell and its child processes. This is what -t does.

    $ ssh -t some_user@some_host bash

You can also fix this by forcing the command to create a pty. For bash, you do this by passing a -i argument. The following command-line should also work:

$ ssh some_user@some_host bash -i

Note, however that in this latter case, the shell cannot access /dev/tty and this results in a warning:

bash: no job control in this shell

The reasons for this are explained here. This may or may not be a problem depending on what you plan to do in the shell but using ssh -t is probably the better option.

(note that you can just pass bash as the command because it should be on the path of the user's default shell anyway)