loop not working when nested?

Question

my code below as you see is clear, the inner loop seems to work, but when nested under another, it doesn't, (no idea why)

Idea is to iterate variables a b with while1 and hand it over to while2, the while2 is waiting for a line in log_file to appear for the case statement to execute.

It does not work with actual process and it's real log_file

#!/bin/bash
bi_log='/var/log/process/process.log'
bi_conf='/etc/process/process.conf'
cat input_file | while
    read a b
do
    while IFS= read -r line; do
        case $line in
            "Processing Data ended" )        echo "Found" &&
                                               echo $a $b >> $bi_conf
                                               ;;
        esac
        sleep 1
        break
    done < <(tail -n0 -f $bi_log)
done

Looks like inner loop workin fine, but maybe something seems wrong with nesting it under outer one. variables a & b are not echoed out to file.

Answer 1

There are several problems with your script, starting with the fact that the tail -n0 -f $bi_log is never going to exit, so the outer loop will never get to the second line of input_file.

Try something more like this:

while read -r a b ; do
  tail -n0 -f "$bi_log" | 
    awk -v a="$a" -v b="$b" \
      '/Processing Data ended/ {
         print "found" > /dev/stderr;
         print a, b ;
         exit
       }' >> "$bi_conf"
done < input_file

Here the awk script prints its output and exits as soon as it finds the match. This terminates the tail -n0 -f pipeline and the shell while loop moves on to the next line of input_file (and starts a new tail | awk pipeline).

BTW, the -v a="$a" -v b="$b" is how you pass shell variable $a into awk variable a and shell variable $b into awk variable b. Alternatively, you could export a b in shell (so that they're exported to the environment visible to child processes) and then use ENVIRON["a"] and ENVIRON["b"] in awk.

If I knew more about input_file and the a and b variables, it's quite likely the entire thing could be done in awk. or perl.

---

Alternatively, you could do it with just bash and grep. e.g.

while read -r a b ; do
  tail -n0 -f "$bi_log" | grep -q -m 1 'Processing Data ended'
  echo "Found"
  echo "$a $b" >> "$bi_conf"
done < input_file

The -m NUM option tells grep to exit after NUM matches (in this case, NUM=1). The -q option tells grep to be quiet, i.e. produce no output and just return an exit code (0 for success, 1 for failure...but because grep's input is coming from tail -f it will keep reading forever until it finds a match).

---

PS: as a general rule of thumb, if you ever finding yourself writing a while/read loop in shell, you should stop and think "I should probably do this in awk/perl/python/anything-but-shell". And then write it in a more appropriate language.

Shell is great for co-ordinating the execution of text and data processing tools. It's lousy at doing the processing itself.

See Why is using a shell loop to process text considered bad practice? for details and examples.