2

I have set up agent forwarding on my ssh client:

Host *
  ForwardAgent yes
  User somebody
  IdentityFile ~/.ssh/id_rsa
  AddKeysToAgent yes

When I log in to the remote system, I see the following in my environment:

bastion1 $ env | grep SSH
SSH_CONNECTION=84.92.91.231 43330 10.10.0.207 22
SSH_AUTH_SOCK=/tmp/ssh-tXaJaakQAi/agent.9736
SSH_CLIENT=84.92.91.231 43330 22
SSH_TTY=/dev/pts/0

However, I also want to use connection sharing:

Host *
  ForwardAgent yes
  User somebody
  IdentityFile ~/.ssh/id_rsa
  AddKeysToAgent yes
  ControlPath ~/.ssh/S.%r@%h:%p
  ControlMaster auto
  ControlPersist yes

This seems to prevent agent forwarding from working - note the missing SSH_AUTH_SOCK:

bastion1 $ env | grep SSH
SSH_CONNECTION=84.92.91.231 42996 10.10.0.207 22
SSH_CLIENT=84.92.91.231 42996 22
SSH_TTY=/dev/pts/0

Is it possible to overcome this problem?

Edit 1

Master connection (please note that some output occurred on this connection as I connected the slave connection - after the bastion1 prompt):

$ ssh -vv rgs-gameiom
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/jan/.ssh/config
debug1: /home/jan/.ssh/config line 46: Applying options for rgs-gameiom
debug1: /home/jan/.ssh/config line 61: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/jan/.ssh/S.jan@35.178.7.91:22" does not exist
debug2: resolving "35.178.7.91" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 35.178.7.91 [35.178.7.91] port 22.
debug1: Connection established.
debug1: identity file /home/jan/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/jan/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 35.178.7.91:22 as 'jan'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DXIktsHBhM8nmqno2LbFZyb7puxygfT8mQbpbGvjyTw
debug1: Host '35.178.7.91' is known and matches the ECDSA host key.
debug1: Found key in /home/jan/.ssh/known_hosts:6
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/jan/.ssh/id_rsa (0x561bdfac2e00), explicit, agent
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:GJMv7wpMeWFOdsaum+toBTWm5C3I+sSkdMrNiWrNszM /home/jan/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:GJMv7wpMeWFOdsaum+toBTWm5C3I+sSkdMrNiWrNszM
Authenticated with partial success.
debug1: Authentications that can continue: password
debug1: Next authentication method: password
jan@35.178.7.91's password: 
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 35.178.7.91 ([35.178.7.91]:22).
debug1: setting up multiplex master socket
debug2: fd 5 setting O_NONBLOCK
debug1: channel 0: new [/home/jan/.ssh/S.jan@35.178.7.91:22]
debug2: fd 3 setting TCP_NODELAY
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 21360
debug2: fd 5 setting O_NONBLOCK
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug1: multiplexing control connection
debug2: fd 6 setting O_NONBLOCK
debug1: channel 1: new [mux-control]
debug2: process_mux_master_hello: channel 1 slave version 4
debug2: mux_client_hello_exchange: master version 4
debug2: process_mux_alive_check: channel 1: alive check
debug2: process_mux_new_session: channel 1: request tty 1, X 0, agent 1, subsys 0, term "xterm-256color", cmd "", env 1
debug1: channel 2: new [client-session]
debug2: process_mux_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 2: request auth-agent-req@openssh.com confirm 0
debug2: client_session2_setup: id 2
debug2: channel 2: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 2: request env confirm 0
debug2: channel 2: request shell confirm 1
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel_input_status_confirm: type 99 id 2
debug2: PTY allocation request accepted on channel 2
debug2: channel 2: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 2
debug2: shell request accepted on channel 2
Linux bastion1.gameiom-production.gamingrealms.org 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Oct  8 13:11:47 2019 from 80.229.227.116
bastion1 $ debug1: multiplexing control connection
debug2: fd 10 setting O_NONBLOCK
debug1: channel 3: new [mux-control]
debug2: process_mux_master_hello: channel 3 slave version 4
debug2: process_mux_alive_check: channel 3: alive check
debug2: process_mux_new_session: channel 3: request tty 1, X 0, agent 1, subsys 0, term "xterm-256color", cmd "", env 1
debug1: channel 4: new [client-session]
debug2: process_mux_new_session: channel_new: 4 linked to control channel 3
debug2: channel 4: send open
debug2: channel_input_open_confirmation: channel 4: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 4: request auth-agent-req@openssh.com confirm 0
debug2: client_session2_setup: id 4
debug2: channel 4: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 4: request env confirm 0
debug2: channel 4: request shell confirm 1
debug2: channel_input_open_confirmation: channel 4: callback done
debug2: channel 4: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 4
debug2: PTY allocation request accepted on channel 4
debug2: channel 4: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 4
debug2: shell request accepted on channel 4

Slave:

$ ssh -vv rgs-gameiom
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/jan/.ssh/config
debug1: /home/jan/.ssh/config line 46: Applying options for rgs-gameiom
debug1: /home/jan/.ssh/config line 61: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug1: mux_client_request_session: master session id: 4
Last login: Tue Oct  8 14:00:12 2019 from 80.229.227.116
j4nd3r53n
  • 715
  • Which SSH client? I have both options set, and both control files and SSH agent work fine for me. (tested with macOS OpenSSH_7.9p1 client, Ubuntu OpenSSH_7.6p1 server) – muru Oct 08 '19 at 08:33
  • @muru - good to hear that it works for you. I use openssh 7.6 client (on Ubuntu 18.04) and on the server it is 7.4 on Debian 9. I wonder if a sysadmin has changed an option on the server? – j4nd3r53n Oct 08 '19 at 09:28
  • @j4nd3r53n I can't think of a server option that would make ForwardAgent incompatible with multiplexing (you can disable agent forwarding on the server, but then it wouldn't work without multiplexing either). Please post transcripts from ssh -vv with a master and a slave connection (redact names and IP addresses that you aren't comfortable sharing). – Gilles 'SO- stop being evil' Oct 08 '19 at 13:51
  • @muru Edit added – j4nd3r53n Oct 08 '19 at 14:05
  • Weird. The trace looks similar to the one I get here (I just tried from Ubuntu 16.04 to Debian stretch) with both agent forwarding and multiplexing enabled. – Gilles 'SO- stop being evil' Oct 08 '19 at 15:31
  • I'm running OpenSSH_7.6p1 and ControlMaster + ForwardAgent randomly breaks. I haven't been able to figure steps to reproduce but it seems to happen more often if I have a long running session open. The problem manifests by remote host not being able to connect to agent. Killing all shared ssh sessions + reconnecting fixes the problem. And I did check that my agent pointed by environment variable SSH_AUTH_SOCK has been running all the time (the process was started 3 days ago). I would guess this is some kind of race condition where ssh permanently loses connection to agent. – Mikko Rantalainen Jul 14 '21 at 11:35

1 Answers1

1

Agent forwarding is compatible with connection sharing. The two interact in that the agent forwarding happens in the master connection only. A slave connection doesn't do agent forwarding, because there'd be no point to forward the agent again if the master has already done it. You'll get the same value of SSH_AUTH_SOCK in all the connections made through the same master connection.

What must have happened here is that you turned on connection sharing, then started a master connection without agent forwarding, either because you had agent forwarding disabled at the time or because you started the master connection from a context where SSH_AUTH_SOCK was not set on the client. Then you made a slave connection attached to that master connection.

Close the master connection and start a new one, making sure that it runs in a context where it has access to the agent (e.g. not a cron job where SSH_AUTH_SOCK is not defined).

Gilles 'SO- stop being evil'
  • 829,060
  • Well, I have this problem right after I've rebooted, actually. I start the agent under systemd as described here: https://unix.stackexchange.com/questions/339840/how-to-start-and-use-ssh-agent-as-systemd-service – j4nd3r53n Oct 08 '19 at 12:05
  • @j4nd3r53n Have you confirmed that SSH_AUTH_SOCK is set on the client where you start the master connection? – Gilles 'SO- stop being evil' Oct 08 '19 at 12:44
  • Yes, it is set to the expected value: /run/user/1000/ssh-agent.socket. Also, it works fine if I comment out the ControlMaster line in ~/.ssh/config – j4nd3r53n Oct 08 '19 at 13:07
  • Keep in mind that many DEs (Gnome/KDE) will start their own ssh-agent, which might not be using all the options you would expect. You can disable their agent by masking it via the ~/.local/applications folder and a .desktop file for gnome-keyring or gnome-settings-daemon that turns off the ssh/gnupg functionality and lets you start it yourself in your .bashrc or .profile. – dragon788 Nov 16 '20 at 21:50