Two arrays in nested for loop over ssh lead to unwanted connection-user permutations

Question

I am trying to update some files on 3 nodes, where I want to combine every connection with a specific user path. As I am very new to this, I extended a for loop I used to connect as root, into a nested for loop with 2 different arrays.

The following script tries of course to use all 3 user paths on every connection. I also tried to use an array only with ips, then every user is trying to connect to all 3 nodes.

#!/bin/bash
EXTENSIONS='/home/bi/.local/share/gnome-shell/extensions'
NODE_SRVS=(
'mf@192.168.122.60'
'lf@192.168.122.92'
'qf@192.168.122.93'
)
NODE_USERS=(
'mf'
'lf'
'qf'
)
echo -e "sync my extensions to my nodes \c"; read
for i in ${NODE_SRVS[@]}; do
  for u in ${NODE_USERS[@]}; do
    rsync -rt --delete --info=progress2,stats2 

    -e ssh -tt 

    $EXTENSIONS/ 

    $i:/home/$u/.local/share/gnome-shell/extensions/
  done
done

Is there a way to restrict the array permutations so that the 1st array connection corresponds only to the first array username, etc?

Else, what is the correct looping construction to match every connection only to the corresponding user path?

cas · Accepted Answer · 2021-12-04T04:00:23.940

Yes, iterate over the indices of the arrays, rather than their values. BTW, The two arrays share indices, so you only need one iterator variable ($i below).

bash arrays start from zero, and you can get the number of elements with "${#arrayname[@]}" - so you need to iterate from zero to one less than that. An array with three elements will have indices 0, 1, and 2 - assuming there were no gaps in the indexing when the array was created. By "gaps", I mean where numeric indices are "missing" (either never inserted into the array or deleted some time after insertion) - e.g. an array might have indices 1, 3, and 5. 2 and 4 are missing.

e.g.

i=0
while [ "$i" -lt "${#NODE_SRVS[@]}" ] ; do
setting $h and $u isn't strictly necessary but makes the code
easier to read, at little performance cost.
h="${NODE_SRVS[$i]}"
   u="${NODE_USERS[$i]}"
rsync -rt --delete --info=progress2,stats2 

    -e ssh -tt 

    "$EXTENSIONS/" 

    "$h:/home/$u/.local/share/gnome-shell/extensions/
let i+=1
done

You can also use "${!arrayname[@]}" to get a list of all the indices - listing only the indices that exist, ignoring any gaps. You can iterate over that with for:

for i in "${!NODE_SRVS[@]}" ; do 
  h=....
  u=...
  rsync ...
done

This is useful for indexed arrays (because of the possibility of gaps), but is especially useful for associative arrays, which brings me to the next alternative: use an associative array (aka "hash"), e.g. where the user names are the keys (indices) and the hostnames/IP addresses are the values (in simple cases like this, it would work just as well to have the IP addresses be the keys and the usernames be the values).

declare -A NODES
NODES=([mf]=192.168.122.60)
NODES+=([lf]=192.168.122.92)
NODES+=([qf]=192.168.122.93)
EXTENSIONS="something"
for u in "${!NODES[@]}"; do 
  rsync -rt --delete --info=progress2,stats2 

    -e ssh -tt 

    "$EXTENSIONS/" 

    "$u@${NODES[$u]}:/home/$u/.local/share/gnome-shell/extensions/"
done

Sample output with an echo at the start of the rsync line:

rsync -rt --delete --info=progress2,stats2 -e ssh -tt something/ qf@192.168.122.93:/home/qf/.local/share/gnome-shell/extensions/
rsync -rt --delete --info=progress2,stats2 -e ssh -tt something/ lf@192.168.122.92:/home/lf/.local/share/gnome-shell/extensions/
rsync -rt --delete --info=progress2,stats2 -e ssh -tt something/ mf@192.168.122.60:/home/mf/.local/share/gnome-shell/extensions/

Note, though, that associative arrays in bash are inherently unordered - that means you'll iterate over the keys in effectively random order. Sometimes this matters, sometimes it doesn't. When it doesn't matter, it's convenient to use an associative array. When it does matter, use two indexed arrays as in the first example above.

When you want the convenience of a hash but need to process the array in the same order it was created, use a hash but also use an indexed array with the keys to the hash as the values of the array. Iterate over the indexed array to get the key names for the hash in the order they were inserted. e.g.

...
ORDER=(mf lf qf)
for u in "${ORDER[@]}"; do
  rsync -rt --delete --info=progress2,stats2 

    -e ssh -tt 

    "$EXTENSIONS/" 

    "$u@${NODES[$u]}:/home/$u/.local/share/gnome-shell/extensions/"

done

Kusalananda · Answer 2 · 2021-12-03T19:00:33.297

To iterate over the paired-up elements of two arrays rather than overall combinations of elements from the arrays, loop over one and pull the next element off of the other in each iteration.

This is made easier (in terms of syntax) if the second list is kept in the list of positional parameters, on which we may use shift to remove elements:

#!/bin/bash
list1=( element1 element2 element3 )
list2=( thing1   thing2   thing3   )
Assign the second list to
the list of positional parameters:
set -- "${list2[@]}"
Loop over the first list and pull out
the next element of the second list
in each iteration, to pair up the elements:
for element in "${list1[@]}"; do
    printf 'Pair: %s %s\n' "$element" "$1"
    shift
done

Output:

Pair: element1 thing1
Pair: element2 thing2
Pair: element3 thing3

Applying this to your specific example:

#!/bin/bash
node_servers=(
    '192.168.122.60'
    '192.168.122.92'
    '192.168.122.93'
)
node_users=(
    'mf'
    'lf'
    'qf'
)
extension_path=.local/share/gnome-shell/extensions
set -- "${node_users[@]}"
for server in "${node_servers[@]}"; do
    rsync -ai --delete ~bi/"$extension_path"/ "$1@$server:$extension_path"
    shift
done

However, since your original server list already contains the usernames, we don't need that second list:

#!/bin/bash
node_servers=(
    'mf@192.168.122.60'
    'lf@192.168.122.92'
    'qf@192.168.122.93'
)
extension_path=.local/share/gnome-shell/extensions
for server in "${node_servers[@]}"; do
    rsync -ai --delete ~bi/"$extension_path"/ "$server:$extension_path"
done

Both code snippets above assume that the current directory will be the home directory of the user we are connecting with.

Alexander Sofianos · Answer 3 · 2021-12-03T18:27:41.233

0

The solution I applied easier from taking into account all your suggestions for which I am thankfull is to use the array indexes like so:

#!/bin/bash
EXTENSIONS='/home/bi/.local/share/gnome-shell/extensions'
IPS=(
'192.168.122.60'
'192.168.122.92'
'192.168.122.93'
)
USERS=(
'mf'
'lf'
'qf'
)
echo -e "sync my extensions to my nodes \c"; read
for i in ${!IPS[*]}; do
  rsync -rt --delete --info=progress2,stats2 

  -e ssh -tt 

  $EXTENSIONS/ 

  ${USERS[$i]}@${IPS[$i]}:/home/${USERS[$i]}/.local/share/gnome-shell/extensions/
done

edited Dec 03 '21 at 18:27

answered Dec 03 '21 at 18:20

Alexander Sofianos

3

You really should double-quote your variables when you use them, including strings containing variable(s) that need to be interpolated, and relying on shell word-splitting with unquoted ${!IPS[*]} instead of quoted "${!IPS[@]}" - these are all bad habits to get into. In short, never trust that a variable will always contain only the kinds of values you think they contain - even if you're 100% certain you control where those variables get their data from, it only takes one bug, one bad value to break your script or execute code that you didn't intend to be executed. – cas Dec 04 '21 at 04:10
Treating data as code is a very common cause of bugs, including security bugs. It's why, for example, the eval keyword is often warned against as a potential source of serious bugs. See Why does my shell script choke on whitespace or other special characters? – cas Dec 04 '21 at 04:12
Thanks for the pointer cas! I 've been wondering why some people double quote variables even though they appear to work fine unquoted. – Alexander Sofianos Dec 04 '21 at 08:16
I actually posted the wrong link - I somehow mangled it to have the right title but the link goes to a different (but related) page. The link I meant to post is: Why does my shell script choke on whitespace or other special characters? – cas Dec 04 '21 at 08:31
I read both threads; they are very educational! – Alexander Sofianos Dec 04 '21 at 13:03

Two arrays in nested for loop over ssh lead to unwanted connection-user permutations

3 Answers3

setting $h and $u isn't strictly necessary but makes the code

easier to read, at little performance cost.

Assign the second list to

the list of positional parameters:

Loop over the first list and pull out

the next element of the second list

in each iteration, to pair up the elements: