3

I would like to run a for-loop, which computes the following task: Run awk 10 times, sum output and print final result. The code is attached below:

sum=0
for i in {1..10}
  do
  count=`awk '{if ($NF==$i) {print $NF}}' * | wc -l`
  sum=$[sum+count]
  echo $sum
done

The issue is, if I change $NF==$i to $NF==1, then the result is correct, but I would like to use the for-loop to run 10 times.

What is the issue in the code?

Paulo Tomé
  • 3,782
9f241e21
  • 133

4 Answers4

7

It would be more efficient to do the whole thing in awk:

awk -v n=0 '
  {for (i = 1; i <= 10; i++) if ($NF == i) {n++; break}}
  END {print n}' ./*

Or if the last field is always an integer:

awk -v n=0 '
  $NF >= 1 && $NF <= 10 {n++}
  END {print n}' ./*

Or if you want the comparison to be lexical and only allow the 1, 2, 3..., 10 representations of those numbers (not 01, 1e0, 1.0...):

awk -v n=0 '
  $NF ~ /^([123456789]|10)$/ {n++}
  END {print n}' ./*
Stéphane Chazelas
  • 544,893
4

The issue here is that you are trying to get shell variable expansion inside single quotes ' ... '. However, inside single quotes, shell variable expansion is suspended (see e.g. this answer here or Lhunath & GreyCat's Bash Guide), which is the reason why this is actually recommended: as the $ performs a similar function in awk in dereferencing individual input fields (but where the field number can be expressed by a variable name as well, as in e.g. $NF), enclosing the program in single quotes avoids concurrent "variable expansions".

In your case, you can "import" the value into the awk program with

awk -v fieldnr="$i" '{if ($NF==fieldnr) {print $NF}}'

Still, it would appear that your problem can be solved entirely in awk, so maybe you want to explain what you want to accomplish in more detail and we can try to find a more elegant (and possibly faster) way; it may be reasonable to open another question, though ...

AdminBee
  • 22,803
  • Thanks for your reply. The intention is to run a for loop, each loop do awk '{if ($NF==$i) {print $NF}}' * | wc -l, where i from 1 to 10. I find that change to count=awk -v fieldnr=$i '{if ($NF==fieldnr) {print $NF}}' * | wc -l does the job. Your solution is very helpful! – 9f241e21 Feb 18 '20 at 13:03
  • 1
    @9f241e21 You don't need a shell loop since awk can do the whole thing on it's own, and even if you did need a loop for some reason you don't need to pipe the awk output to wc since awk can count lines for itself too and given that you don't need awk to print $NF. Also, awk is made up of condition { action } statements so you shouldn't put an explicit condition inside the action part while leaving an implicit true condition to execute it. You got an answer to the question you asked but if you'd like help to do whatever it is you're trying to do the right way then post a new question. – Ed Morton Feb 18 '20 at 18:02
  • 1
    @EdMorton Thanks for your reply. I agree that my solution is very inefficient. I'll check based on your suggestions – 9f241e21 Feb 19 '20 at 00:55
3

If your script does what I think it does (i.e. print incremental values of sum for the matching lines as it loops from 1 to 10) then all you really need to produce the output your shell script would produce is:

awk '
{ counts[$NF]++ }
END {
    for (i=1; i<=10; i++) {
        sum += counts[i]
        print sum
    }
}
' *

The above is untested since you didn't provide any sample input/output to test against.

Ed Morton
  • 31,617
-1
#!/bin/bash
# Tested with -- GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
set -e

for i in `seq 1 1000`
do
  echo $i
  # Run your command here
done
marikhu
  • 1
  • You haven't shown them the problem with their code or how to run awk in there. Notice the previous answer that pointed out the problem with single quotes preventing variable expansion. Notice also that the OP appears to be able to create a for loop construct in bash, using brace expansion. Also note that the loop asked for 10 iterations, not 1000. – Jeff Schaller Apr 07 '20 at 12:23