0
sudo tshark -i ppp0 'tcp port 80   \
and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'   \
-R'http.request.method == "GET" && http.request.uri contains "/ABC/XYZ"' \
-T fields -e http.host -e http.request.uri

I am using the above lines of tshark filtering the output based on /ABC/XYZ. it outputs dynamically random data for example like where i find duplicate lines example :

1bcdJOSHhijklmnopqrstuvwxyz
1bcdefghijklmnopqrstuvwxyz
1bcdefghijklmnopqrstuvwxyz
3bcdefghijklmnopqrstuvwxyz
2bcdefghijklmnopqrstuvwxyz
1bcdJOSHhijklmnopqrstuvwxyz
3bcdefghijklmnopqrstuvwxyz

Is there any way to remove the duplicates without creating any file may be using some kind of buffer or pipe.

or can tshark itself do it

EDIT : I am not sure why but i get what i want after i prefix tshark with stdbuf -o L as suggested by meuh

munish
  • 7,987

1 Answers1

3

If you have a stop condition on your tshark you can simply pipe the output into |sort -u. Alternatively, pipe continuous output into

awk '{if(!seen[$0]++)print}'

You may need to have tshark not buffer its stdout: try prefixing the tshark with stdbuf -o L.

meuh
  • 51,383
  • do you know why stdbuf -o L helps here – munish Apr 02 '16 at 10:14
  • 1
    Some programs use the standard library to print data and the library detects if the output is to the terminal or to a file or pipe. If output is to a terminal, each print is output immediately, but if to a file the output is optimised and only written when there is a full buffer. stdbuf is a utility to make the standard library never buffer output. – meuh Apr 02 '16 at 10:41