Compare awk vs. grep

Question

I have run below two commands for a very large files

grep -E 'string1|string2' 151103*.log|grep 'string3' | grep string4

awk '/string1|string2/ && /string3/ && /string4/' 151103*.log

It took almost same time for execution. But awk was much faster to show me the results which matched. grep too showed me the same result but at the end, when the process completed.

Both took same time for the process to complete, just want to know the logic behind both the searches for awk and grep.

Why is awk faster? Do both programs have different search logic? What if I jumble the strings in above search, does it make a difference to the search speed?

Answer 1

GNU grep buffers output but GNU awk does not. And even if you were not using GNU awk and were using some other variant it would likely still be line-buffered if you were printing to a terminal and so would flush output for each occurring \newline, but your grep writes to a pipe and so would block-buffer anyway. If you have a GNU grep you can use grep --line-buffered ... | grep ... for your comparison to see results as quickly. Likely grep will beat awk at practically any match tests - especially a GNU grep.

Here is a sed to do what you want as well:

sed -ne'/string4/{/string3/s/string[12]/&/p;}' <in >out

Answer 2

The grep pipeline could not output anything until the final grep for string4 matched something, and it only gets its input after the previous pipe buffer fills up. See related questions How big is the pipe buffer? and Turn off buffering in pipe .

Depending on the frequency of the strings in your input, you could see a difference in runtimes by putting the static searches first, to give the extended regexp's less to look at.

Answer 3

Your awk example is doing the entire regex search in one pass. For each line of input, if the first, second, and third regex is found, the line will be printed, and you'll see the output essentially immediately (upon processing of the matching line).

Your grep example is using 3 different invocations of grep (one for each regex) to do the same thing, but the output of each invocation becomes the input for the next, which means that each needs to complete before the next has anything to process.

If you had a single 1000 line file and only line 5 matched all three regex, the awk command would give you output after processing the 5th line, before processing the 6th line. Compare that to the piped grep statements. The 1st invocation of grep would find the 5th line and whatever other lines may match the 1st regex, and after processing the 1000th (final) line of input, its output becomes the input to the 2nd invocation of grep. The second invocation of grep processes however many lines the 1st output and outputs the lines that match both the 1st and 2nd regex, which then becomes the input to the 3rd invocation of grep. As the 3rd invocation of grep processes each line, it will output any line that matches its regex.

You can compare the best and worst cases of grep for the above example: if none of the lines match any of the regex except line 5, which matches all 5, then the first grep processes 1000 lines, the second grep processes 1 line, and the third grep processes 1 line: it will process 1002 lines before has any output (best case). If all the lines match the first two regex but only one line matches the third regex, then the piped grep construction will process 1000 + 1000 lines + 5 = 2005 lines before it finds the match on the 5th line and has some output (it will continue to process the remaining 995 lines from the second grep's output, but you won't see any more output because nothing else will match).

Compare that to the awk command, which checks all three regex simultaneously for each line and gives you output after processing the 5th line. The difference will be exaggerated as you check more files simultaneously.

For instance, compare if you see output faster if instead of running the grep command on all files simultaneously as you did above (theoretically, you should, but results may vary depending on distribution of hits throughout your files):

grep -E 'string1|string2' 151103*.log|grep 'string3' | grep string4

you instead run the series of grep commands on each file individually, like this:

for i in 151103*.log; 
  do grep -E 'string1|string2' $i |grep 'string3' | grep string4; 
done

This still won't produce output as quickly as the awk statement, but you may see a difference.

Answer 4

While grep and awk and sed can be used for similar tasks each has its strengths and weaknesses.

Awk works best for tabularised data or when you need to perform calculations etc.

Sed excells in replacing text.

Grep is best to select rows from input data so I was expecting that to be faster than awk for this task. Perhaps if you combine the 3 grep commands into one that's what you'll see. Right now grep is at the disadvantage since it needs to start 3 times and the second and third need to wait for input from the first. Which might explain why the result comes with a delay. Although I'm not sure about that.