How to run for loop over SSH on remote so that variables expand?

Question

If I ls files without a for loop in a remote host everything is fine however, if I capture the ls in a variable and try to echo each file it fails as the variables don't expand/have a value.

What I mean:

IFS=$'\n'
servers=(
  blue
  green
  red
)

for i in ${servers[@]}; do 
   ssh $i "
     ls /dir/xyz_${i}*/details.txt
   "
done
/dir/xyx_blue0/details.txt 
/dir/xyz_blue1/details.txt
/dir/xyx_green2/details.txt 
/dir/xyz_green4/details.txt
/dir/xyx_red1/details.txt 
/dir/xyz_red8/details.txt

But I actually need to loop through the output of ls so I can do things to the files, however the variable doesn't expand:

for i in ${servers[@]}; do 
   ssh $i "
     for i in $(ls /dir/xyz_${i}*/details.txt); do 
        echo $i
     done
   "
done
not found: /dir/xyx_blue*/details.txt 
not found: /dir/xyx_green*/details.txt 
not found: /dir/xyx_red*/details.txt 

How can I get $i to expand when running a loop on the remote host?

Answer 1

The issue is that the command substitution that you want to run on the remote servers is run locally. This is due to the substitution occurring within double quotes, meaning the local shell will compute its expansion before calling ssh.

Also, reading the output of ls is ill-advised (see Why *not* parse `ls` (and what to do instead)?).

Instead:

servers=(
  blue
  green
  red
)
for server in "${servers[@]}"; do
    ssh -T "$server" <<END_SCRIPT
shopt -s nullglob
for pathname in /dir/xyz_$server*/details.txt; do
    printf 'file path is "%s"\n' "$pathname"
done
END_SCRIPT
done

This uses an unquoted "here-document" as the script that is sent to the remote servers. The script first sets the nullglob shell option (this is assuming that the remote shell is bash) to avoid looping at all in the inner loop if the pattern does not match any filenames. It then iterates over the pattern /dir/xyz_$server*/details.txt where $server will be substituted with the current server name.

The loop then prints out a short message about what filenames matched the pattern, and the code ensures that the $pathname variable is not expanded by the local shell by escaping the $.

Note that $ in $server is unescaped and will therefore be expanded by the local shell, which is what we want, but that $ in $pathname needs to be escaped to supress the local expansion.

We don't need to set IFS to anything non-default. I'm assuming you did that to be able to write the array assignment as you did, or for some other reason, but none of the code above needs that.

I'm using -T with ssh to explicitly turn off pseudo-TTY allocation.

---

You could also provide the script as a single string if you feel that this is somehow needed:

servers=(
  blue
  green
  red
)
for server in "${servers[@]}"; do
    ssh "$server" "
shopt -s nullglob
for pathname in /dir/xyz_$server*/details.txt; do
    printf 'file path is "%s"\n' "$pathname"
done"
done

In this case, you must additionally ensure that each embedded double quote is escaped (or they would end the double-quoted string that constitutes the remote shell script).

Answer 2

Never for-loop over the output of ls; that's literally what * globbing is for, and your whole usage of ls is totally superfluous.

That solves the whole problem, honestly, because you're currently running the $(…) locally instead of on the target host.