12

I'm able to run this command successfully:

tail -f my_file.txt | grep foo

It shows only the lines with the string foo, and it keeps showing them.

But when I run this command:

tail -f my_file.txt | grep foo | grep bar

It doesn't show any lines, even though there are lines that include both foo and bar.

I know there is a solution for using multiple patterns in a single grep call, but I want to know why this line failed.

Ram Rachum
  • 1,825

2 Answers2

27

That's because the default behaviour of the C runtime library is to buffer writes to stdout until a full block of data is written (some kilobytes, usually), unless stdout is connected to a terminal.

You'll get output once the middle grep has printed a full block, but then you have to wait again for the next block to fill, and so on. It's an optimization for throughput, and works much better when the left-hand command just does some task and terminates, instead of waiting for something.

GNU grep has the --line-buffered option to turn off that buffering, so this should work better:

tail -f my_file.txt | grep --line-buffered foo | grep bar

The last grep prints to the terminal so it's line buffered by default and doesn't need an option.

See Turn off buffering in pipe for generic solutions to the buffering issue.

In this particular case of two greps, you could use e.g. a single AWK instead as Stéphane Chazelas mentioned in a comment:

tail -f my_file.txt | awk '/foo/ && /bar/'

(Incidentally, you could also do things like awk '/foo/ && !/bar/', catching lines with foo but no bar.)

Doing the same in grep would be harder, as grep -e foo -e bar matches any lines that contain either foo or bar. You'd need something like

... | grep -E -e 'foo.*bar|bar.*foo'

instead.

ilkkachu
  • 138,973
-2

From a boolean point of views it seems you're expecting foo OR bar with your grep, yet from the way you did you should expect foo AND bar -- The only lines that will ever be grepped for bar are those that passed the foo grep in the first place.

If you want either foo/bar popping up on stdout you need to use:

tail -f my_file.txt | grep "foo|bar"

You can add as many keywords "pretty|much|like|these", just don't forget the quotes. (Or use multiple\|unquoted\|terms)

On a second note: Since it's a tail -f (follow) it might also happen for your_file.txt to not be appended with a line containing both keywords at that very specific time, it's usually best to cut a share of that file that is known to have what you expect:

tail -n 500 my_file.txt > my_sample.txt

From there you can try whatever grep you feel like with a known text. (I'm guessing the last 500 lines is enough, adjust as needed.)

Carlos Lint
  • 9
  • 1
    For foo OR bar, that's either grep -e foo -e bar or grep -E 'foo|bar'. Basic regexps (without -E) don't have an alternation operator though some grep support it as a \| extension. – Stéphane Chazelas Dec 24 '22 at 08:38
  • 3
    The OP make it very clear that they want lines that include both foo and bar, so they want AND, not OR. – Stéphane Chazelas Dec 24 '22 at 08:39
  • 1
    tail -f file prints the last 10 lines that were initially in the file and waits and prints all the additional lines that come thereafter. To print all the initial lines and all following, use tail -n +1 -f file. – Stéphane Chazelas Dec 24 '22 at 08:42