How can I store all the lines emitted by while loop in a single varianle in shell?

Question

As far as I know, while loops in shell are executed in a sub-shell, so they cannot modify variables outside the loop.

I'm writing a shell script and I want to store all the internal IPs of my machine into a single variable and so process this variable with a for loop to filter them one by one with iptables.

I could write this piece of code:

ip route show default | awk '{ print $5 }' | while read line; do
  ip address show dev ${line} scope global | awk '/inet / {sub(/\/.*/, "", $2); print $2}' | while read line; do
    echo "${line} "
  done
done

Output:

10.17.0.49 
192.168.1.4

My question is:

How can I store all these lines emitted by a while loop into a single variable (as while loop variables are volatile)?

It's almost never necessary to store data in intermediate variables. Are you not wanting to use the IP addresses fro further processing? If so, then why not do that processing in your loop? Also, what do you mean by "volatile"? You mean "local to the subshell"? — Kusalananda, Oct 01 '22 at 05:42
Related: Why is using a shell loop to process text considered bad practice? — Stéphane Chazelas, Oct 01 '22 at 05:53
@Kusalananda Yes, I want to use the IP addresses for further processing. I think I will have to processes in a loop then. Volatile, I mean "local to the subshell". — phantomcraft, Oct 02 '22 at 01:33

Stéphane Chazelas · Answer 1 · 2022-10-01T07:08:07.273

To print all the global scope local addresses of the interfaces involved in default routes, I'd use the JSON format which can be processed programmatically in a more reliable way:

perl -MJSON -le '
  $default_routes = decode_json(qx(ip -j route show default));
  for (@$default_routes) {$devs{$_->{dev}} = 1}
  $addresses = decode_json(qx(ip -j address show));
  for (@$addresses) {
    if ($devs{$_->{ifname}}) {
      for (@{$_->{addr_info}}) {
        print $_->{local} if $_->{scope} eq "global";
      }
    }
  }'

Or the same using jq:

ip -j address show |
  jq -r --argjson devs "$(
      ip -j route show default|jq 'map({"key":.dev})|from_entries'
    )" '.[]|select(.ifname|in($devs)).addr_info[]|
      select(.scope == "global").local'

(it needs a relatively recent version of iproute2 though for JSON output support).

To get it (or more generally every line of the output of some command) into a bash array, use:

readarray -t array < <(
  that-command above
)

If the aim is to get the source IP address that packets going out on the default route would get, see for instance my answer to How to get my own IP address and save it to a variable in a shell script?.

score 2 · Answer 2 · answered Oct 01 '22 at 06:27

To answer succinctly the actual question posed irrespective of context of how to store the output of a while loop:

To store the output into a file, e. g. /path/to/file:

while command; do
    thing
done > /path/to/file

To store the output into a variable, e. g. my_var:

my_var="$(while command; do thing; done)"

Edgar Magallon · Accepted Answer · 2022-10-03T16:39:33.757

1

I know two ways of doing that (in bash):

Using < <(command) called process substitution (this does not pass the content through a pipe to the while loop ).
Using { code or commands } called command grouping.

So if you want to store all IP in a variable (or array in this case) you can use this code:

Process substitution:

process_subs () {
   local array=()
   while read -r line; do
     while read -r line; do
       array+=("$line")
     done < <(ip address show dev "${line}" scope global | awk '/inet / {sub(/\/.*/, "", $2); print $2}')
   done < <(ip route show default | awk '{ print $5 }')
echo Array: "${array[@]}"
   echo Length: ${#array[@]}
}

Command grouping:

command_grouping () {
ip route show default | awk '{ print $5 }' |  while read -r line; do
ip address show dev "${line}" scope global | awk '/inet / {sub(//.*/, "", $2); print $2}' | { 
         local array=()
            while read -r line; do
            array+=("$line")
         done
  echo Array: &quot;${array[@]}&quot;
  echo Length: ${#array[@]}
  }
  done 

}

edited Oct 03 '22 at 16:39

answered Oct 01 '22 at 04:40

Edgar Magallon

4,820

First solution looks in line with what I would suggest myself. Perhaps I'm missing something though because I don't see how your "command grouping" solution addresses the issue. The input to while read still comes via pipe, thus any variable setup will live and die within the loop. – bxm Oct 01 '22 at 05:27
@bxm yeah, the command grouping doesn't really resolve the problem of the variables inside a while but it's a possible solution to make use of them. The downside of that method is that you have to define all your code that depends on the variables inside the command grouping. – Edgar Magallon Oct 01 '22 at 05:51
1

@EdMorton thanks for the advice! I will update the code – Edgar Magallon Oct 03 '22 at 16:38

How can I store all the lines emitted by while loop in a single varianle in shell?

3 Answers3