A way to see what command exactly sshd executes?

Question

I was thinking of passing an environment variable to a command over ssh:

$ ssh HOST A=1 set | grep ^A
$ ssh HOST A=1 sh -c set | grep ^A
A='1'
$ ssh HOST A=1 echo '$A'
$ ssh HOST A=1 sh -c 'echo $A'

So, set doesn't see the environment variable, but it does in a separate process. The same doesn't seem to happen with echo.

Is there a way to find out what exactly is sent, and how exactly it's executed (which command sshd launches)?

Answer 1

When you run ssh user@host command, the tokens resulting from the expansion of command are concatenated by ssh with a space in between and sent as a string to the remote host. The remote host then executes an instance of the remote user's default shell (the one specified in /etc/passwd), passing the string it received from the client as the argument to the shell's -c option (more on this below).

So, for instance, in

$ foo="bar    baz"
$ ssh user@host echo "$foo"
bar baz

the local ssh receives the two arguments echo and bar baz. It concatenates them to form the string echo bar baz and sends it to the server. The server executes something along the lines of /bin/sh -c 'echo bar baz' and the output of this command is printed on the local ssh's standard output.

How can you see what the client actually sends to the server?

As Kamil Maciorowski pointed out in a comment to the question, ssh -v outputs as a debug message the command string being sent to the server:

$ ssh -v host A=1 sh -c set | grep ^A
...
debug1: Sending command: A=1 sh -c set
...

How can you see what sshd actually executes?

I am not aware of any way to request the server to show the client some kind of trace of its own operations. But, for testing purposes, you can strace an instance of the daemon:

$ sudo strace -e execve -f /usr/bin/sshd -p 2222

This is especially useful because it is easy not to get the quoting right when composing a command line for remote execution. Take, for instance (a variation on the examples provided in your question):

$ ssh -p 2222 host bash -c 'A=1 set'
                                        # Prints nothing

strace reveals that what we may see as a command string is actually split on the remote host, which first runs the full command line in the user's default shell (the single quotes have been removed by the local shell when it built the argument list for ssh):

... execve("/bin/bash", ["bash", "-c", "bash -c A=1 set"] ...

and then:

... execve("/usr/bin/bash", ["bash", "-c", "A=1", "set"] ...

Note how the command string (the argument to -c) happens to be just A=1. set is passed to bash as the zeroth argument (available as $0 and used as the command name).

How to see what the remote shell actually executes?

Again, I am not aware of any easy way. In simple cases, the use of the shell's xtrace option is straightforward. For instance, ssh host A=1 set may become:

$ command=$(printf '%s ' A=1 set); printf '%s\n' "${command% }" |
  ssh host sh -x

sh should be replaced with the remote user's default shell.

The command is piped to ssh's standard input because this form allows for typing it unchanged (ssh host sh -xc A=1 set won't work for the reason shown above and adding quoting would defeat, to some extent, the very purpose of this exercise).

In less simple cases, especially when the the ssh invocation you are digging into already makes use of standard input, you may force the remote host to run a shell with xtrace turned on. For experimenting purposes, and having clear that errors may make you unable to log in to the remote system, a (rather sketchy) way could be to replace the remote user's default shell in /etc/passwd. Assuming it was /bin/sh, change it into /home/your_user/bin/sh and create the latter as an executable script:

#!/bin/sh
exec /bin/sh -x "$@"

Which will make the shell show you a trace of what is being executed:

$ ssh host A=1 echo foo
foo
+ A=1
+ echo foo

Finally, about your sample commands:

• The reason ssh HOST A=1 set doesn't print A=1 is likely that your user's default shell is Bash (possibly invoked as sh). Contrary to what happens in dash, ksh, yash, zsh, set in Bash (at least the version I have, 5.0.17) does not output the variables set for the set command itself. On the other hand, you could probably confirm that:

  $ ssh host 'A=1
  > set' | grep ^A
  A=1
  

  I.e. A is shown in set's output if it is declared before set is invoked.

• In A=1 sh -c set, A is added to the environment of sh and is thus unsurprisingly printed by set.

• A=1 echo '$A' doesn't print the value of A because A is only added to the environment for the execution of echo (assuming it is built-in), after the expansion of $A. Compare it with ssh host 'A=1; echo $A', which prints 1.

• A=1 sh -c 'echo $A' prints nothing because, as shown above, the remote host executes the remote user's default shell with the -c option and A=1 sh -c echo $A as its argument, and then that shell runs sh -c echo $A: the command string ends up being just echo, with no arguments.

  To make it work you need to make sure the quotes around echo $A are sent to the remote host and $A is protected from being expanded too early (by the local shell or by the "outer" remote one): e.g. ssh host A=1 sh -c '"echo \$A"' or ssh host A=1 sh -c "'echo \$A'".

  When explicitly calling a remote shell, it is probably easier to make it read commands from standard input:

  $ echo 'echo $A' | ssh host A=1 sh -s
  1
  

See also

• How do ssh remote command line arguments get parsed
• Quoting in ssh $host $FOO and ssh $host "sudo su user -c $FOO" type constructs