Using grep in a variable in a loop

Question

I'm having trouble storing a grep result as a variable in a loop.

while read file;do
  Server=$(echo $file | awk '{ print $1 }')
  FDate=$(echo $file | awk '{ print $2 }')
  ST=$(cat foobar | grep $Server | awk '{ print $3 }')
  #ST=$(grep $Server foobar | awk '{ print $3 }')
echo "Server = $Server"
  echo "FDate = $FDate"
  echo "ST = $ST"
done < inputfile

The first ST var gives the output "Usage: grep [Option]... Pattern [File]" for each iteration which means its not reading the command correctly.

The second ST var that is commented out actually breaks the entire script cause all the other variables to be empty when it tries to echo.

Now when I try doing the same thing on the command line it works:

$ testme=$(cat foobar | grep Big | awk '{ print $3}'
$ echo "$testme"
tada

So my question is how do I store that grep command in the variable? The pattern match has only one possible result so I don't have to worry about multiple matches. But each server in the loop might have a different string in column 3 (tada,tada1,tada2)

EDIT:

The inputfile has a list of servers with multiple columns. I'm taking the server listed in column 1 of that current line and searching the foobar file for a match and getting the string from column 3.

I've found that the script actually does work even though it's giving the 'Usage' message. Probably because some of the server entries in the inputfile aren't yet in the foobar file so grep doesn't have a match but still tried to pipe it to awk. I don't know that for certain.

I'd still like to eliminate the 'Usage' messages though. I think maybe a 'set -o pipefail' might work but I'd rather not do that.

Answer 1

while read file;do                                       # 1
  Server=$(echo $file | awk '{ print $1 }')              # 2
  FDate=$(echo $file | awk '{ print $2 }')               # 3
  ST=$(cat foobar | grep $Server | awk '{ print $3 }')   # 4
  #ST=$(grep $Server foobar | awk '{ print $3 }')        # 5

grep needs at least the pattern to search for (or an -e or -f option providing the equivalent), so if $Server ends up being empty, then the unquoted $Server on lines 4 and 5 disappear during word splitting (see also When is double-quoting necessary?), and

• the grep on line 4 gets no arguments. Without the mandatory argument, it prints the usage description.
• the grep on line 5 gets the single argument foobar, which it takes as a pattern. By default it reads from standard input, and inside the loop, it has the same stdin as the loop, so eats everything from there.

---

Now, the whole loop reminds me of this question: Why is using a shell loop to process text considered bad practice? and it could be simplified at least somewhat. read can split the input on fields itself, so we can remove the command substitutions.

Then, we should probably deal with the case where one or both of the values happen to be empty. And, since awk can do the job of grep, too, let's do that:

while read server fdate; do
    if [ -z "$server" ] || [ -z "$fdate" ]; do
        continue
    fi
    ST=$(awk < foobar -v server="$server" '$0 ~ server { print $3 }')
echo &quot;server $server fdate $fdate ST $ST&quot;

done < inputfile

(or, depending on what you're intending to do in the end, replace the whole thing with an awk program.)

Answer 2

Don't do this with grep and/or awk in a shell while loop (or any commands like that in a shell loop, for that matter). See Why is using a shell loop to process text considered bad practice? for reasons why. In short: shell is great at getting other programs to do work, setting up redirections and pipelines and feeding filenames and data into other programs that actually do the work, but it is terrible at doing that work itself. Shell is slow and prone to user errors like not double-quoting your variables (e.g. your failure to quote $Server as "$Server" is the direct cause of your problem with grep...one of the causes, anyway. The other cause is your failure to check whether $Server actually contained any value after the awk)

Anyway, everything you need can be done with one short awk script. For example:

awk 'NR==FNR { fdates[$1] = $2 ; next}; # read first file into fdates array
 $1 in fdates {  # process second file
   printf "Server = %s\nFDate = %s\nST = %s\n", $1, fdates[$1], $3;
 }' inputfile foobar

In English:

• Read the first file, store it in an associative array called fdates, with the key being field 1 (Server) and the value being field 2 (FDate)

• When we're reading the second file (and any subsequent files), if the server name is a key in fdates, then print out the details you wanted.

NOTE: You haven't specified which field you expect to find the server name in in file foobar. The script above assumes that it is in field 1 of foobar. If it's in a different field, change the $1 in two lines (the $1 in fdates line and the printf line) to suit.

If the server names could be anywhere on a line in foobar (i.e. there is no fixed field number to match on) then you could write the script like this:

awk 'NR==FNR { fdates[$1] = $2 ; next};
     { for (server in fdates) {
       if ($0 ~ server) {
         printf "Server = %s\nFDate = %s\nST = %s\n", server, fdates[server], $3;
       }
     }' inputfile foobar

In English:

• Read the first file, store it in an associative array called fdates, with the key being field 1 (Server) and the value being field 2 (FDate) - i.e. same as the first version.

• Then, for each and every line of subsequent file(s), iterate over every element of fdates array. If there's a regex match with the key (server name) anywhere on the line, then print your required details.

The second version will be a bit slower than the first because it has to do a regex match for every server name contained in the fdates array for each line of foobar.

Both versions will be orders of magnitude faster than your while read loop around 3 calls to awk, and 1 call each to cat and grep - reading the entirety of file foobar on each pass through the loop. Either of the awk scripts above is only called once, and they only have to read inputfile and foobar once.