5

I've got a log file that I process as follows:

grep pattern /var/log/whatever.log | \
    cut ... | sort | uniq -c | sort -rn | \
    etc....

However, the log file is quite big, and records events from the beginning of the day. It's also appended to constantly. I'd like to only process the last next 10 minutes of the file.

So I'm looking for something like the following:

killafter 600 tail -f /var/log/whatever.log | stuff

Or, better yet (wait however long it takes to capture 1000 matching lines):

tail -f /var/log/whatever.log | grep pattern | stopafter -lines 1000 | stuff

Are there any tools that'll let me do this?

Roger Lipscombe
  • 1,700

5 Answers5

8

roaima rightly points you at the timeout command, and head actually terminates after it has read the desired number of lines, so I would hope that with

timeout 600s tail -f ‹logfile› | ‹filter› | head -n 1000

you'd get there.

Ulrich Schwarz
  • 15,989
  • That is a good answer I had difficulty with head never seems to take stdin properly. Try timeout 600s tail -f /var/log/whatever.log | grep -m 1000 pattern – Willtech Feb 26 '22 at 01:49
1

By default, timeout kills the process and the pipe gets killed.

timeout --foreground 600s tail -n 0 -f logfile | ...filter | head -n 1000| ...
  • timeout --foreground otherwise we would see just "Terminated"
  • tail -n 0 -f to show only the new loglines
  • head -n 1000 to "stop after 1000 lines"

\thanks{roaima and Ulrich Schwarz}

JJoao
  • 12,170
  • 1
  • 23
  • 45
  • 2
    How is this different to Ulrich Schwarz's answer? – Chris Davies Sep 09 '16 at 10:16
  • @roaima, Ulrich Schwartz's answer, gets "terminated" and prints no output in case of timeout. – JJoao Sep 09 '16 at 10:36
  • @roaima and Ulrich, unfortunately my answer is not working properly too: in case of obtaining "1000 lines", it prints the answer but the "timout" process continues :( – JJoao Sep 09 '16 at 10:41
0

This perl code will give you a counted tail

#!/usr/bin/perl
#
# Usage tailmax.pl <filename> [<num_lines>]
#
use strict;
use warnings;

use File::Tail;

my $logfile = shift @ARGV;
my $pattern = shift @ARGV || '';
my $count = shift @ARGV || -1;

my $fh = File::Tail->new(name => $logfile, tail => 0, interval => 0, maxinterval => 2) or
    die "Cannot open $logfile: $!\n";

while ($count != 0 and defined (my $line = $fh->read)) {
    if ($pattern eq '' or $line =~ /$pattern/) {
        print "$line";
        $count-- if $count > 0;
    }
}

If you save it to a file such as /usr/local/bin/tailmax and make it executable, you can call it like this, which will give you the next 100 messages from the log file /var/log/messages that contain the RE somepattern. (You may need to be root to read this file.)

tailmax /var/log/messages 'somepattern' 100
Chris Davies
  • 116,213
  • 16
  • 160
  • 287
0

Tried with below command and it worked fine

tail -f /var/log/file.log & read -t 600; kill $!

This would run tail -f on the file in the background, and then use read with a timeout set to 10 minutes. When the read returns, either because the user pressed Enter, or because of timing out, the background tail -f job is killed.

Kusalananda
  • 333,661
Praveen Kumar BS
  • 5,211
0

I had difficulty with the answer from Ulrich Schwarz because head never seems to look at stdin so try the following if it may be of assistance.

timeout 600s tail -f {logfile} | grep -m 1000 {pattern}

Good Luck!

Willtech
  • 101