How to tail -f against a file which is rolled every 500MB / daily?

Question

I would like to do the following to monitor for a pattern in a log file:

tail -f ./app.log | grep "My Pattern: .*" >> ./MyPattern.txt &

This is working fine until the app.log reaches 500MB and it is moved to be app-2015-10-28.0.log and a new app.log is created. When this happens the tail -f will stop working. Stopping and running the same command again will work, but in between stopping and starting, the pattern that appears in between the restarting may be missed.

Another condition of rolling is when it is 00:00. The same rolling will happen even when the log is < 500MB.

Log rolling is done at application level and we cannot control that part.

How can we detect the log has been rolled, without missing the patterns of interest?

tail -F or tail --follow=name will follow the filename rather than the file descriptor. — Mark Wagner, Oct 27 '15 at 22:35
Dupe of http://unix.stackexchange.com/questions/22698/how-to-do-a-tail-f-of-log-rotated-files BTW — Mark Wagner, Oct 27 '15 at 22:37

score 3 · Accepted Answer · edited Oct 27 '15 at 23:14

tail normally follows the file descriptor (--follow=descriptor), which becomes inaccessible when the log file is rolled or the file is moved away. See this from the man page of tail:

--retry

keep trying to open a file even when it is or becomes inaccessible; useful when following by name, i.e., with --follow=name

So use it like this (tail follows the name not the descriptor now):

tail --follow=name --retry logfile | grep "pattern"

You will see in tails output, when the file gets truncated or moved away:

$ tail --follow=name --retry logfile
[...]
tail: logfile: file truncated
[...]
tail: `logfile' has become inaccessible: No such file or directory
tail: `logfile' has appeared;  following end of new file
[...]

How to tail -f against a file which is rolled every 500MB / daily?

1 Answers1