0

I got this project to analysis my logs where I need to find any consecutive errors occurred more then thrice and list logs name. As my logs were huge, I wrote below script:

for i in `ls`  
  do 
  count=`uniq -c $i | grep 'FATAL ERROR' |sed 's/^ *//g'| sed 's/ /,/g' | awk -F',' '{if($1 >1) {print $1}}'`  
  if [[ $count -ge 2 ]]  
  then  
    echo $i  
  fi  
done

This works if you just need to know number of occurrence in which all file 

grep -o -c Source * | awk -F: '{if ($2 > 2){print $1}}'

What could have been the better version of my script? What if log is really huge, my piece of code will slowdown.

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
Machine
  • 99
  • 2
  • 8

1 Answers1

2

Pipelines should generally be kept short, and most of the things that you do to get the value for count could be done in a single awk program, including the loop and the if statement.

awk 'FNR == 1 || !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/              { ++count }
     count == 2                 { print FILENAME }' ./*

This runs a single awk program across all non-hidden files in the current directory.

The program will reset its count variable to zero if we're at the first line of a new file or if the current line does not match the pattern FATAL ERROR.

If the line matches the pattern FATAL ERROR, the count variable is incremented.

If the count variable reaches a value of 2, the name of the current file is printed.

The code will print the name of the file each time it finds two consecutive lines that matches the pattern, even if this happens multiple times in the same file. If this in not wanted, you could expand the code slightly:

awk 'FNR == 1       { count = 0; do_print = 1 }
     !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/  { ++count }
     count == 2 && do_print { print FILENAME; do_print = 0 }' ./*

or with GNU awk (using nextfile to skip to the next file):

awk 'FNR == 1 || !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/              { ++count }
     count == 2 && do_print     { print FILENAME; nextfile }' ./*

Related to your shell loop:

Kusalananda
  • 333,661