Can't process stdout with pipe as it comes

Question

I'm running tshark on a fifo, and the following is a bare example of a loop that prints the output of tshark as it comes:

tshark -i $fifo | while read line; do
    echo $line
done

The problem appears when I add filters to tshark. This example prints all $lines only after tshark exits (IP address is hidden):

tshark -i $fifo -T fields -e text -R '
    ip.src == **.**.***.**          &&
    http.response.code == 200       &&
    http.content_encoding == "gzip" &&
    http.content_type contains "text/html"
' | while read line; do
    echo $line
done

I have tried in other forms with no luck:

while read line; do
    echo $line
done < <(tshark ...)

while read line; do
    echo $line
done <<<"$(tshark ...)"

Even grep prints the lines only after tshark ends:

tshark ... | grep .

I have tried running tshark without a pipe and the lines are printed correctly as they come. Why is the command after the pipe being feeded only after tshark exits?

Additional details: | tee works, but I get everything printed again when tshark exits, so it is not a good deal.

Hard to believe that a program's output shall depend on the program reading from the pipe... I would expect a buffering problem. But how should the receiving end influence this? Stupid as it sounds: Maybe tshark checks what's at the other side of the pipe? strace is your friend. What about cat (instead of tee)? And why is the output printed twice? — Hauke Laging, Apr 14 '13 at 23:54
@HaukeLaging Definitely a buffering problem: unbuffer tshark worked! I read the man page for unbuffer, but I couldn't really understand the reasoning behind this... see: http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe — admirabilis, Apr 15 '13 at 00:22

admirabilis · Answer 1 · 2013-04-15T13:50:36.360

5

I was able to get it working with stdbuf from coreutils. Note that every command after the pipe requires the buffer to be adjusted too:

stdbuf -o 0 tshark -i $fifo -T fields -e text -R '
    ip.src == **.**.***.**          &&
    http.response.code == 200       &&
    http.content_encoding == "gzip" &&
    http.content_type contains "text/html"
' | 
stdbuf -o 0 sed 's/\\r\\n,\?/\n/g; s/\\t/\t/g' |

From the man page:

`stdbuf': Run a command with modified I/O stream buffering

(...)

`-o MODE'
`--output=MODE'
     Adjust the standard output stream buffering.

edited Apr 15 '13 at 13:50

answered Apr 15 '13 at 02:17

admirabilis

4,712

Why should every command after the pipe need adjustment, too? – Hauke Laging Apr 15 '13 at 03:27
@HaukeLaging Otherwise the output may get truncated. I got it truncated on the 80th line. – admirabilis Apr 15 '13 at 04:25
Great trick, but stdio buffering exists for a good reason, and it's lame that utilities which mess with stuff like isatty() force you to do this. – Cera Apr 15 '13 at 05:19

score 4 · Accepted Answer · answered Apr 15 '13 at 14:21

4

Check if your tshark version has the -l option for (nearly) line-buffered output.

answered Apr 15 '13 at 14:21

vro

56

1

Right, this is the fix on the tshark part. Although, as I have shown in my own answer, if more than one pipe is going to be used, the others will have to be fixed too. As a simple example, tshark -l ... | grep . | grep . doesn't work, whereas tshark -l ... | stdbuf -o 0 grep . | grep . works, so all information on this thread was useful. Thanks everybody so much for all the help! – admirabilis Apr 15 '13 at 15:20

score 3 · Answer 3 · answered Apr 15 '13 at 01:06

Some utilities call isatty() to determine whether their output is a terminal and adjust their behaviour accordingly. gzip is a good example of this.

Try running it with script(1):

-c, --command command

Run the command rather than an interactive shell. This makes it easy for a script to capture the output of a program that behaves differently when its stdout is not a tty.

If you run it like this:

script -c tshark -i $fifo -T fields -e text -R '
    ip.src == **.**.***.**          &&
    http.response.code == 200       &&
    http.content_encoding == "gzip" &&
    http.content_type contains "text/html"
' | while read line; do
    echo $line
done

You should be able to see lines come out live.

Thanks, but script requires something more. I get script: invalid option -- 'i' — admirabilis, Apr 15 '13 at 02:13
Yo may need to put the entire command in quotes. Gets tricky with such a complex command. Try putting your entire call to tshark in a shell script, and then doing script -c myscript.sh. — Cera, Apr 15 '13 at 05:17

Can't process stdout with pipe as it comes

3 Answers3