How does ssh pick which key to use?

Question

I'm using the following as a forced command with hg-ssh.

This forced command is designed to just allow pushes with hg. However, I must be doing something wrong, because I can actually log in with this key. I've used this for a while, and my recollection is that it used to exit immediately if I used this key to try to log in.

I have this following in .ssh/authorized_keys.

command="cd /srv/hg/faheem && find . -mindepth 1 -maxdepth 4 -type d -exec /usr/bin/hg-ssh {} +",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa public_key

This may have an obvious answer, but I don't even know what the shell command

cd /srv/hg/faheem && find . -mindepth 1 -maxdepth 4 -type d -exec /usr/bin/hg-ssh {} +

is doing, exactly.

Stack Exchange automatically produced this related question that I asked in 2011 and forgot about, so it looks like I got this command from this answer of @Gilles.

UPDATE: I have two keys for the same local-remote combination. One of these is set up as a regular key in ~/.ssh/authorized_keys. I.e. it looks like

ssh-rsa public_key

The other is as given above. The regular key is listed first, and the forced-command key second in ~/.ssh/authorized_keys. From the output of ssh -vvv the "regular" key is being used. Here are the relevant stanzas client-side in .ssh/config.

Host ramnode
Hostname xx.x.xxx.xxx
ForwardX11 yes

Host ramnode_hg
Hostname xx.x.xxx.xxx
ForwardX11 yes
identityfile ~/.ssh/id_rsa_bulldog_hg

The first is the "regular" key, the second is the forced-command key.

Perhaps it is not possible to specify which key should be used, and that ssh just tries keys in order in ~/.ssh/authorized_keys till something works.

UPDATE 2: Having tried switching the order of the keys in ~/.ssh/authorized_keys, with the forced-command key first, this appears to not be the case. Doing

ssh ramnode_hg

still logs me in to the machine.

UPDATE 3: The output of ssh -vvv when logging into the machine.

OpenSSH_6.0p1 Debian-4+deb7u1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/faheem/.ssh/config
debug1: /home/faheem/.ssh/config line 122: Applying options for ramnode_hg
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to xx.x.xxx.xxx [xx.x.xxx.xxx] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/faheem/.ssh/id_rsa_bulldog_hg" as a RSA1 public key
debug1: identity file /home/faheem/.ssh/id_rsa_bulldog_hg type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/faheem/.ssh/id_rsa_bulldog_hg-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4+deb7u1
debug1: match: OpenSSH_6.0p1 Debian-4+deb7u1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "xx.x.xxx.xxx" from file "/home/faheem/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/faheem/.ssh/known_hosts:68
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 4e:66:01:8f:a1:44:f6:d9:01:e4:72:ff:6a:ab:8e:31
debug3: load_hostkeys: loading entries for host "xx.x.xxx.xxx" from file "/home/faheem/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/faheem/.ssh/known_hosts:68
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'xx.x.xxx.xxx' is known and matches the ECDSA host key.
debug1: Found key in /home/faheem/.ssh/known_hosts:68
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/faheem/.ssh/id_rsa (0x7f81f0ca2550)
debug2: key: /home/faheem/.ssh/id_rsa_bb (0x7f81f0ca23b0)
debug2: key: /home/faheem/.ssh/id_rsa_bulldog_hg (0x7f81f0c9d590)
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/faheem/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: fp a1:78:e9:c1:f4:9c:5e:23:a3:d9:38:35:be:0a:b4:8b
debug3: sign_and_send_pubkey: RSA a1:78:e9:c1:f4:9c:5e:23:a3:d9:38:35:be:0a:b4:8b
debug1: Authentication succeeded (publickey).
Authenticated to xx.x.xxx.xxx ([xx.x.xxx.xxx]:22).
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: x11_get_proto: /usr/bin/xauth  list :0 2>/dev/null
debug1: Requesting X11 forwarding with authentication spoofing.
debug2: channel 0: request x11-req confirm 1
debug2: client_session2_setup: id 0
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env FULLNAME
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env KDE_MULTIHEAD
debug3: Ignored env DM_CONTROL
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env XDM_MANAGED
debug3: Ignored env XDG_SESSION_COOKIE
debug3: Ignored env GTK2_RC_FILES
debug3: Ignored env KONSOLE_DBUS_SERVICE
debug3: Ignored env CVSROOT
debug3: Ignored env GS_LIB
debug3: Ignored env GTK_RC_FILES
debug3: Ignored env WINDOWID
debug3: Ignored env OLDPWD
debug3: Ignored env SHELL_SESSION_ID
debug3: Ignored env KDE_FULL_SESSION
debug3: Ignored env HISTFILESIZE
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env COLUMNS
debug3: Ignored env PATH
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PWD
debug3: Ignored env EDITOR
debug3: Ignored env PSQL_EDITOR
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env KDE_SESSION_UID
debug3: Ignored env PYTHONSTARTUP
debug3: Ignored env KONSOLE_DBUS_SESSION
debug3: Ignored env HISTCONTROL
debug3: Ignored env GPG_TTY
debug3: Ignored env CVS_SSH
debug3: Ignored env SHLVL
debug3: Ignored env COLORFGBG
debug3: Ignored env HOME
debug3: Ignored env KDE_SESSION_VERSION
debug3: Ignored env LANGUAGE
debug3: Ignored env XCURSOR_THEME
debug3: Ignored env LOGNAME
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env LESSOPEN
debug3: Ignored env LVM_SUPPRESS_FD_WARNINGS
debug3: Ignored env EMAIL
debug3: Ignored env PROMPT_COMMAND
debug3: Ignored env WINDOWPATH
debug3: Ignored env ALTERNATE_EDITOR
debug3: Ignored env DISPLAY
debug3: Ignored env PROFILEHOME
debug3: Ignored env QT_PLUGIN_PATH
debug3: Ignored env RSYNC_RSH
debug3: Ignored env LESSCLOSE
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: X11 forwarding request accepted on channel 0
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Linux faheem 2.6.32-042stab079.6 #1 SMP Mon Aug 26 19:47:50 MSK 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Apr 10 01:43:43 2014 from [localhost]

UPDATE 4: I have an ssh agent running, and ssh-add -L confirms it has keys loaded.

The command executes hg-ssh, passing files under /srv/hg/faheem as arguments. Is that the only line in .ssh/authorized_keys? Does public_key stand for a base64 string that starts with AAAA? — Gilles 'SO- stop being evil', Apr 09 '14 at 22:42
Hi Gilles. No, that is not the only line in .ssh/authorized_keys. There is one other, but that is not a forced-command key. Yes, the public key in question does start with AAAA. What is the significance of that? — Faheem Mitha, Apr 09 '14 at 22:48
Just checking that the file is well-formed. Is the AAAA… part different in the two lines? — Gilles 'SO- stop being evil', Apr 09 '14 at 22:50
Yes, the two public key strings are different for the two lines. — Faheem Mitha, Apr 09 '14 at 23:36
Removing the non-forced-command key/line from .ssh/authorized_keys makes the forced-command key behave correctly. — Faheem Mitha, Apr 09 '14 at 23:42
Are you sure you aren't sending the non-forced-command key? Do you use ssh-agent or similar? Post the trace from ssh -vv …. — Gilles 'SO- stop being evil', Apr 09 '14 at 23:45
It looks like it is using the non-forced-command key. Further comments in the question. — Faheem Mitha, Apr 10 '14 at 12:26

derobert · Accepted Answer · 2014-04-10T22:01:32.630

The client picks which key to send. In fact, you can see that in your verbose client output:

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/faheem/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 277

The client offered to prove its identity with that key (the client picked this key because its the first one in its list of default key locations for protocol 2 that exists, the full list is in the ssh(1) manpage), so the server checked the authorized_keys file. It found the key is authorized, so it said OK. Then the client went ahead and proved its identity with the key:

debug2: input_userauth_pk_ok: fp a1:78:e9:c1:f4:9c:5e:23:a3:d9:38:35:be:0a:b4:8b
debug3: sign_and_send_pubkey: RSA a1:78:e9:c1:f4:9c:5e:23:a3:d9:38:35:be:0a:b4:8b
debug1: Authentication succeeded (publickey).
Authenticated to xx.x.xxx.xxx ([xx.x.xxx.xxx]:22).

By default, when you specify IdentityFile, it just gives files to search in addition to anything already loading in ssh-agent (which you can check by running ssh-add -l or ssh-add -L). To tell it to only use the files you specify, you must also add IdentitiesOnly yes to your config.

How does ssh pick which key to use?

1 Answers1