I have a log file that contains the following content.
2021-06-15T22:50:11+00:00 DEBUG {"slug": "something", "key2": "value2"}
I would like to tail -f this file and pipe the results to jq command, but I need to strip out 2021-06-15T22:50:11+00:00 DEBUG part before piping to jq since jq expects a JSON string.
Is there a way to tail the log file and strip the datetime part at the same time?
Ultimately, I would like to use the following command.
tail -f :file | jq
Assuming you have access to GNU sed which is able to do unbuffered output:
tail -f file | sed -u 's/^[^{]*//' | jq .
This would run tail -f on your file and continuously send new data to sed. The sed command would strip everything up to the space before the first { on the line, and then send the result on to jq.
The -u option to GNU sed makes it not buffer the output. Without this option, sed would buffer the result and would only send data to jq once the buffer (4 Kb?) was full. Doing buffering like this is standard procedure when the output of a tool is not the terminal itself, and it's done for efficiency reasons. In this case, we may want to turn the buffering off, so we use -u.
To select only lines that contain the DEBUG string before the JSON data:
tail -f file | sed -u -e '/^[^{]*DEBUG /!d' -e 's///' | jq .
or
tail -f file | sed -u -n 's/^[^{]*DEBUG //p' | jq .
The sed command here would delete all lines that do not start with some text not containing { characters, ending in DEBUG . If such a line is found, the matched text is removed, leaving the JSON data.
Note that we here extract the JSON based on the DEBUG string rather than the { that initiates a JSON object.
Related to buffering in pipelines:
To remove the first 2 space delimited columns:
tail -f file | stdbuf -oL cut -d ' ' -f3- | jq .
(stdbuf -oL, as found on GNU or FreeBSD systems, tricks cut into doing line-based instead of block-based output buffering).