extract fields from "tail -f" of a syslog stream

Question

Example line from syslog file:
Aug  1 10:25:50 10.10.10.1 id=firewall sn=XXXX time="2012-08-01 14:35:18 UTC" fw=x.x.x.x pri=6 c=1024 m=537 msg="Connection Closed" f=11 n=195273698 src=x.x.x.x:60246:X3: dst=x.x.x.x:443:X6:host.domain.com proto=tcp/https sent=181793 rcvd=649678

I want to do a

$ tail -f /var/log/log.log | SOMETHING

That will return

"src=x.x.x.x dst=x.x.x.x"

In plainspeak I want to be able to extract specific fields from text. I do not want to select these fields based on a field number (position in the line) because the order of things can change slightly with the system generating this file, so it doesn't work.

Open to suggestions but would prefer it to be a one-liner.

score 4 · Answer 1 · answered Aug 01 '12 at 14:46

4

tail -f log | perl -e 'while (<>) { print "src=$1 dst=$2\n" if $_ =~ /src=([^\s]+) dst=([^\s]+)/; }'

answered Aug 01 '12 at 14:46

daisy

54,555

I like it, and am intrigued by perl. Care to share a little bit of what's going on there? (trying to learn) – LVLAaron Aug 01 '12 at 17:59
@LVLAaron, it try to match "src=XXX dst=YYY" on each line, where XXX and YYY are fields that contains no spaces or tabs, if got any match, it print matched part – daisy Aug 01 '12 at 23:42

score 3 · Answer 2 · edited Apr 13 '17 at 12:36

3

sed 's/^.*\(src=[0-9.][0-9.]*\).*\(dst=[0-9.][0-9.]*\).*$/\1 \2/'

See Extracting a regex matched with 'sed' without printing the surrounding characters

edited Apr 13 '17 at 12:36

Community

1

answered Aug 02 '12 at 00:26

Gilles 'SO- stop being evil'

829,060

score 1 · Answer 3 · edited Oct 25 '13 at 14:05

1

grep has a --line-buffered option which allows you to follow tail -f

tail -f /var/log/messages | grep --line-buffered  'src=[0-9.]+\s+dst=[0-9.]+'

edited Oct 25 '13 at 14:05

Anthon

79,293

answered Aug 01 '12 at 23:50

Hermes Conrad

119

This returns nothing. – LVLAaron Aug 03 '12 at 14:38

Brandon Kreisel · Answer 4 · 2012-08-01T17:30:24.883

0

not quite as pretty but another solution implemented with awk:

tail -f /var/log/log.log | awk 'FS="src=" { print "src="$2 }' | awk 'FS=":" { print $1$4 }'`

Update: edited to be one line for tail -f

edited Aug 01 '12 at 17:30

answered Aug 01 '12 at 16:59

Brandon Kreisel

1,098

thank you @jw013. I didn't catch that. I revised it to a one liner. – Brandon Kreisel Aug 01 '12 at 17:31

score 0 · Answer 5 · answered Aug 02 '12 at 14:08

0

Or, improving a little on warlOck's:

tail -f log | perl -n -e 'print "$1 $2\n" if /(src=[^\s]+) (dst=[^\s]+)/;'

(I haven't figured out how to comment on an answer yet...)

answered Aug 02 '12 at 14:08

sebnukem

166

extract fields from "tail -f" of a syslog stream

5 Answers5