Why does echoing $? vary differently when running in this script?

Question

I have the following in my script:

connectionTest=$(sshpass -p "${pass}" ssh -q -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "${remoteUser}"@"${IP}" "echo quit | netcat -w 5 local.server.local ${telnetPort}; echo $?")

When I take the command out and run it manually on the machine, I get the exit code I'm expecting. However, whenever the script runs, the resulting exit code is always 0.

Why would the exit code always return as 0?

Answer 1

The reason you're getting an unexpected return status is the way you have quoted your command for the remote host.

Consider and contrast these two statements:

a=apple; ssh remotehost "a=banana; echo $a"
a=apple; ssh remotehost 'a=banana; echo $a'

In the first case, the "... $a" is evaluated before the ssh command is run, leading to this effective situation:

ssh remotehost "a=banana; echo apple"

In the second case, the '... $a' is passed as a literal to the remote host for execution, leading to this effective situation:

ssh remotehost 'a=banana; echo $a'

This is exactly what is happening with your $?. Because you have enclosed it with double-quotes, it's evaluated before the command is run. Its value of zero is from your previous command, so what you are effectively running is something like this:

connectionTest=$(sshpass -p PASSWORD ssh ... "REMOTEUSER@REMOTEIP "echo quit | netcat ... ; echo 0")

What you need is to use single quotes to have the local shell treat it as a literal, like this:

connectionTest=$(sshpass -p "${pass}" ssh ... "${remoteUser}@${IP}" "echo quit | netcat -w 5 local.server.local ${telnetPort}; "'echo $?')

Notice that a string of two parts such as "hello"'world' is concatenated to be a single value helloworld. The first part was subject to variable evaluation; the second part was not.