66

I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following:

tail -f /path/to/serverLog | grep "server is up" ...(now, e.g., wget on server)?

What is the best way to do this?

jonderry
  • 2,089
  • 4
    be sure to use tail -F to handle log rotation - i.e. my.log becomes full and moves to my.log.1 and your process creates a new my.log – s g Apr 24 '15 at 22:42
  • Here's a related discussion in the Ubuntu Q&A site: https://askubuntu.com/q/1052891/31592 – blong Mar 12 '21 at 18:41

7 Answers7

42

A simple way would be awk.

tail -f /path/to/serverLog | awk '
                    /Printer is on fire!/ { system("shutdown -h now") }
                    /new USB high speed/  { system("echo \"New USB\" | mail admin") }'

And yes, both of those are real messages from a kernel log. Perl might be a little more elegant to use for this and can also replace the need for tail. If using perl, it will look something like this:

open(my $fd, "<", "/path/to/serverLog") or die "Can't open log";
while(1) {
    if(eof $fd) {
        sleep 1;
        $fd->clearerr;
        next;
    }
    my $line = <$fd>;
    chomp($line);
    if($line =~ /Printer is on fire!/) {
        system("shutdown -h now");
    } elsif($line =~ /new USB high speed/) {
        system("echo \"New USB\" | mail admin");
    }
}
penguin359
  • 12,077
  • I like the awk solution for being short and easy to do on the fly in one line. However, if the command I want to run has quotes, this can be a bit cumbersome, is there an alternative, maybe using pipelines and compound commands that also allows a brief one-liner solution but doesn't require the resulting command to be passed as a string? – jonderry Apr 26 '11 at 22:39
  • 1
    @jon You can write awk as a script. Use "#!/usr/bin awk -f" as the first line of the script. This will eliminate the need for the outer single quotes in my example and free them for use inside a system() command. – penguin359 Apr 26 '11 at 22:43
  • @penguin359, True, but it'd still be interesting to do it from the command line as well. In my case, there are a variety of different things I'd want to do including many things I can't foresee, so it's convenient to be able to just start the server and do it all in one line. – jonderry Apr 26 '11 at 22:50
  • I found an alternative, though I don't know how solid it is: tail -f /path/to/serverLog | grep "server is up" | head -1 && do_some_command – jonderry Apr 26 '11 at 23:20
  • @jon That seems a little fragile using head that way. More importantly, it's not repeatable like my examples. If "server is up" is in the last ten lines of the log, it will fire the command and exit immediately. If you restart it, it will most likely fire and exit again unless ten lines not containing "server is "up have been added to the log. A modification of that that might work better is tail -n 0 -f /path/to/serverLog That will read the last 0 lines of the file, then wait for more lines to print. – penguin359 Apr 27 '11 at 03:07
  • However, if your goal is to fire a command every single time something appears in the log files, both version of tail and head may miss lines between each invocation. – penguin359 Apr 27 '11 at 03:08
  • Thanks, -n 0 will help. In my current case, a little fragility isn't an issue because I'm just going through experimental iteration. From another question, I found that grep -m 1 "server is up" <(tail -n 0 -f /path/to/serverLog); some_command avoids problems with a pipe remaining alive when I expect it to die and have the command terminate. – jonderry Apr 27 '11 at 18:36
  • @jon You may stil want && instead of ;. The && means execute the next command if and only if the previous command was successful. The ; means execute the second command when the first command finishes. For example, if I hit Ctrl-Z I get a funny behavior and the second command executes immediately. With && this doesn't happen. – penguin359 Apr 27 '11 at 22:02
  • Syntaxe tail -f ... | perl -ne ... may present same simple syntax than using ... | awk .... The most important part of your script is tail-into-perl. – F. Hauri - Give Up GitHub Jul 22 '13 at 17:09
  • Use tail -F (capital f) to account for rotating logs. i.e. my.log becomes full and moves to my.log.1 and your process creates a new my.log – s g Apr 24 '15 at 22:43
31

If you're only looking for one possibility and want to stay mostly in the shell rather than using awk or perl, you could do something like:

tail -F /path/to/serverLog | 
grep --line-buffered 'server is up' | 
while read ; do my_command ; done

...which will run my_command every time "server is up" appears in the log file. For multiple possibilities, you could maybe drop the grep and instead use a case within the while.

The capital -F tells tail to watch for the log file to be rotated; i.e. if the current file gets renamed and another file with the same name takes its place, tail will switch over to the new file.

The --line-buffered option tells grep to flush its buffer after every line; otherwise, my_command may not be reached in a timely fashion (assuming the logs have reasonably sized lines).

agc
  • 7,223
Jander
  • 16,682
  • 3
    I really like this answer, but it didn't work for me at first. I think you need to add the --line-buffered option to grep, or otherwise make sure it flushes its output between lines: otherwise, it just hangs, and my_command is never reached.

    If you prefer ack, it has a --flush flag; if you prefer ag, try wrapping with stdbuf. http://stackoverflow.com/questions/28982518/what-is-ags-silver-searcher-analogous-option-to-greps-line-buffered

    – doctaphred Sep 13 '15 at 20:40
  • I tried this, using do exit ;. It seemed to work fine, but tail never finished and our script never moved onto the next line. Is there a way to stop tail in the do section? – Machtyn Feb 28 '19 at 19:08
  • If you need the log line in my_command, you can use ... while read line; do my_command $line ; done – ki9 Mar 22 '22 at 18:21
  • add -n0 to the tail command to ensure it doesn't read any previous lines, ie, after a restart, or conversely -n +0 to ensure it reads from the beginning of the file. – Otheus Nov 12 '23 at 15:53
15

It is strange that no one mentioned about multitail utility which has this functionality out-of-box. One of usage example:

Show the output of a ping-command and if it displays a timeout, send a message to all users currently logged in

multitail -ex timeout "echo timeout | wall" -l "ping 192.168.0.1"

See also another examples of multitail usage.

Graeme
  • 34,027
Slava Semushin
  • 807
  • 7
  • 8
  • +1, I had no idea `multitail had those kind of ninja skills tucked away. Thanks for pointing that out. – Caleb Apr 27 '11 at 21:54
14

This question appears to be answered already, but I think there's a better solution.

Rather than tail | whatever, I think what you really want is swatch. Swatch is a program designed explicitly for doing what you're asking, watching a log file and executing actions based on log lines. Using tail|foo will require that you've got a terminal actively running to do this. Swatch on the other hand runs as a daemon and will always be watching your logs. Swatch is available in all Linux distros,

I encourage you to try it out. While you can pound a nail in with the back side of a screwdriver does not mean you should.

The best 30-second tutorial on swatch I could find is here.

Stephen Kitt
  • 434,908
bahamat
  • 39,666
  • 4
  • 75
  • 104
8

could do the job by himself

Let see how simple and readable it could be:

mylog() {
    echo >>/path/to/myscriptLog "$@"
}

while read line;do
    case "$line" in
        *"Printer on fire"* )
            mylog Halting immediately
            shutdown -h now
            ;;
        *DHCPREQUEST* )
            [[ "$line" =~ DHCPREQUEST\ for\ ([^\ ]*)\  ]]
            mylog Incomming or refresh for ${BASH_REMATCH[1]}
            $HOME/SomethingWithNewClient ${BASH_REMATCH[1]}
            ;;
        * )
            mylog "untrapped entry: $line"
            ;;
    esac
  done < <(tail -f /path/to/logfile)

While you don't use bash's regex, this could stay very quick!

But + is a very efficient and interesting tandem

But for high load server, and as I like sed because it's very quick and very scalable, I often use this:

while read event target lost ; do
    case $event in
        NEW )
            ip2int $target intTarget
            ((count[intTarget]++))
        ...

    esac
done < <(tail -f /path/logfile | sed -une '
  s/^.*New incom.*from ip \([0-9.]\+\) .*$/NEW \1/p;
  s/^.*Auth.*ip \([0-9.]\+\) failed./FAIL \1/p;
  ...
')
6

That's how i started doing this too but have become much more sophisticated with it. A couple things to be concerned with:

  1. If the tail of the log already contains "server is up".
  2. Automatically ending the tail process once it's found.

I use something along the lines of this:

RELEASE=/tmp/${RANDOM}$$
(
  trap 'false' 1
  trap "rm -f ${RELEASE}" 0
  while ! [ -s ${RELEASE} ]; do sleep 3; done
  # You can put code here if you want to do something
  # once the grep succeeds.
) & wait_pid=$!
tail --pid=${wait_pid} -F /path/to/serverLog \
| sed "1,10d" \
| grep "server is up" > ${RELEASE}

It works by holding tail open until the ${RELEASE} file contains data.

Once the grep succeeds it:

  1. writes the output to ${RELEASE} which will
  2. terminate the ${wait_pid} process to
  3. exit the tail

Note: The sed can be more sophisticated to actually determine the number of lines tail will produce at startup and the remove that number. But generally, it's 10.

nix
  • 561
0

I use fail2ban to monitor and execute a command

Jordi
  • 1
  • Perhaps a small example would benefit the OP & others? The question starts with: tail -f /path/to/serverLog | grep "server is up" ... and then execute a command. – Jeff Schaller Oct 15 '21 at 19:34