How to process/pipe TCPDUMPs output in realtime

Question

If I want to tcpdump DNS requests by clients (on an OpenWrt 10.04 router), then I

root@ROUTER:/etc# tcpdump -n -i br-lan dst port 53 2>&1       
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 96 bytes
22:29:38.989412 IP 192.168.1.200.55919 > 192.168.1.1.53: 5697+ A? foo.org. (25)
22:29:39.538981 IP 192.168.1.200.60071 > 192.168.1.1.53: 17481+ PTR? 150.33.87.208.in-addr.arpa. (44)
^C
2 packets captured
3 packets received by filter
0 packets dropped by kernel

That's fully ok. But. Why can't I pipe the tcpdumps output in realtime?

root@ROUTER:/etc# tcpdump -n -i br-lan dst port 53 2>&1 | awk '/\?/ {print $3}'
^C
root@ROUTER:/etc#

If I awk, etc. anything after tcpdump, I don't get ANY output. Why is that? Why can't I process the output of tcpdump with pipelining in realtime? (so that e.g.: in the example in only outputs the 3rd column)

Are there any solutions for this?

score 48 · Accepted Answer · edited Jul 18 '18 at 18:15

48

Straight out of man tcpdump

-l     Make stdout line buffered.  Useful if you want to see the data while 
       capturing it.  E.g.,

              tcpdump -l | tee dat

       or

              tcpdump -l > dat & tail -f dat

       Note that on Windows,``line buffered'' means ``unbuffered'', so that 
       WinDump will write each character individually if -l is specified.

       -U is similar to -l in its behavior, but it will cause output to be 
       ``packet-buffered'', so that the output is written to stdout at the 
       end of each packet rather than at the end of each line; this is 
       buffered on all platforms, including Windows.

edited Jul 18 '18 at 18:15

slm

369,824

answered Jul 11 '11 at 20:42

Marcin

1,597

1

Unfortunately, this doesn't seem to work with tcpdump version 4.9.3 compiled with libpcap version 1.9.1 (with TPACKET_V3). The output seems to still be buffered and I get delayed+buffered output. According to man tcpdump this supports both -l and --immediate-mode but in reality it doesn't seem to work even with both of those added. – Mikko Rantalainen Aug 06 '20 at 08:43
1

It turns out that --immediate-mode -l -n works correctly with tcpdump version 4.9.3. If your DNS server cannot cope with the packet flow your output will be delayed unless you pass -n in addition to -l or --immediate-mode because without the -n the output will be buffered until DNS resolution has been completed or timeout occurs. – Mikko Rantalainen Aug 06 '20 at 09:30

score 9 · Answer 2 · edited Sep 09 '15 at 07:52

9

Use the option -U in combination with -w so that tcpdump writes packets immediately.

edited Sep 09 '15 at 07:52

dr_

29,602

answered Sep 09 '15 at 07:45

Julius Junghans

101

score 3 · Answer 3 · answered Jul 02 '11 at 21:28

The tcpdump is apparently buffering output when it writes to a pipe. It's not flushing output for each write, so the system will write the output in about 4k byte chunks. Your filter is limiting out put so you won't see anything until that filter has written enough output. Once it collects enough it will be written out in a chunk and you should see several lines emitted then.

Try to trigger DNS lookups many times and see what happens then.

score 3 · Answer 4 · answered Apr 13 '18 at 21:12

I'm building a real-time monitoring wrapper around tcpdump that needs to see packets as soon as they are available. Even with -l there is some delay.

tcpdump now has --immediate-mode, which solved this problem for me. In order to get it to work I used it in conjunction with -l.

See this answer.

score 2 · Answer 5 · answered Jul 10 '11 at 21:02

2

expect has an unbuffer command to fool commands into assuming they are writing to a tty, so they don't buffer.

answered Jul 10 '11 at 21:02

ninjalj

1,427

How to process/pipe TCPDUMPs output in realtime

5 Answers5

Linked