How to print streaming JSON to file after processing with JQ and SED

Question

I am using the following code to download streaming financial data

curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output '[.time, .bids[0].price, .asks[0].price] | @csv'

which is successfully streaming in the terminal thus

"2020-05-05T10:02:37.060299264Z","1.08472","1.08481"

However, what I would really like is to further process this, using sed, to get

2020,05,05,10,02,37.060299264,1.08472,1.08481

and then append this to a file, let's call it "output," but when I try piping to sed I no longer see the streaming data although I'm sure my sed syntax is correct as I've checked it against a static test file.

So my question is: how do I further pipe to sed and then append to the file "output?"

Use stdbuf, unbuffer or similar (https://unix.stackexchange.com/a/25378/70524) and see if you get output — muru, May 05 '20 at 11:27
@babelproofreader: jq itself has an option to unbuffer with --unbuffered flag — Inian, May 05 '20 at 11:35
A combination of stdbuf, --unbuffered flag in jq and -u flag in sed accomplishes what I want. Thanks to muru and Inian. — babelproofreader, May 05 '20 at 12:53
@babelproofreader, please post an answer, otherwise this question has no value for future readers. — glenn jackman, May 05 '20 at 16:20
@muru It does. In my posted answer I used the jq --unbuffered flag after it was suggested by Inian — babelproofreader, May 12 '20 at 12:27

babelproofreader · Accepted Answer · 2020-05-05T17:11:45.200

In response to Glenn Jackman's comment requesting that I post an answer, the syntax that I find works for me is:

stdbuf -oL -eL curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output --unbuffered '[.time, .bids[0].price, .asks[0].price] | @csv' | sed -u 's/["Z]//g' | sed -u 's/[-T:]/,/g' >> ~/path/to/append/to/output

which appends to the file in the desired format described in my original question. I can also look at the "output" file from time to time and see that the streaming data is continually being appended.

How to print streaming JSON to file after processing with JQ and SED

1 Answers1