Using jq within pipe chain produces no output

Question

The issue of jq needing an explicit filter when the output is redirected is discussed all over the web. But I'm unable to redirect output if jq is part of a pipe chain, even when an explicit filter is in use.

Consider:

touch in.txt
tail -f in.txt | jq '.f1'
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

As expected, the output in the original terminal from the jq command is:

1
3

But if I add any sort of redirection or piping to the end of the jq command, the output goes silent:

rm in.txt
touch in.txt
tail -f in.txt | jq '.f1' | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

No output appears in the first terminal and out.txt is empty.

I've tried hundreds of variations but it's an elusive issue. The only workaround I've found, as discovered through mosquitto_sub and The Things Network (which was where I also discovered the issue), is to wrap the tail and jq functions in a shell script:

#!/bin/bash
tail -f $1 | while IFS='' read line; do
echo $line | jq '.f1'
done

Then:

./tail_and_jq.sh | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

And sure enough, the output appears:

1
3

This is with the latest jq installed via Homebrew:

$ echo $SHELL
/bin/bash
$ jq --version
jq-1.5
$ brew install jq
Warning: jq 1.5_3 is already installed and up-to-date

Is this a (largely undocumented) bug in jq or with my understanding of pipe chains?

Answer 1

The output from jq is buffered when its standard output is not a terminal.

To request that jq flushes its output buffer after every object, use its --unbuffered option, e.g.

tail -f in.txt | jq --unbuffered '.f1' | tee out.txt

From the jq manual:

--unbuffered

Flush the output after each JSON object is printed (useful if you're piping a slow data source into jq and piping jq's output elsewhere).

Answer 2

What you're seeing here is the C stdio buffering in action. It will store output on a buffer until it reaches a certain limit (might be 512 bytes, or 4KB or larger) and then send that all at once.

This buffering gets disabled automatically if stdout is connected to a terminal, but when it's connected to a pipe (such as in your case), it will enable this buffering behavior.

The usual way to disable/control buffering is using the setvbuf() function (see this answer for more details), but that would need to be done in the source code of jq itself, so maybe not something practical for you...

There's a workaround... (A hack, one might say.) There's a program called "unbuffer", that's distributed with "expect" that can create a pseudo-terminal and connect that to a program. So, even though jq will still be writing to a pipe, it will think it's writing to a terminal, and the buffering effect will be disabled.

Install the "expect" package, which should come with "unbuffer", if you don't already have it... For instance, on Debian (or Ubuntu):

$ sudo apt-get install expect

Then you can use this command:

$ tail -f in.txt | unbuffer -p jq '.f1' | tee out.txt

See also this answer for some more details on "unbuffer", and you can find a man page here too.