1

I have bash buffering a problem similar to what can be found here: Turn off buffering in pipe

The socat solution in the above question is quite interesting as I have access to this command in my initrd dracut hook script, unfortunately I don't see how to apply it to my particular problem: parse journalctl json output in "real time" (avoid buffering; process each line when journalctl output it (every \n).

the logmessage() function is only called when buffer is full; not when a new journal line is printed. I don't know if the problem is before jq or if it is in the "< <(" redirection.

(logmessage() is a simple function that writes the message into a log file and to my Plymouth console).

In the code below, the buffer is processed only when it is full which is of no use for me.

{
        SEVERITY=( emerg alert crit err warning notice info debug )
        FACILITY=( kern user mail daemon auth syslog lpr news uucp cron authpriv ftp ntp security console cron local0 local1 local2 local3 local4 local5 local6 local7 )
        while read LOG_FACILITY LOG_SEVERITY LOG_TAG LOG_MESSAGE
        do
                logmessage ${FACILITY[$LOG_FACILITY]}.${SEVERITY[$LOG_SEVERITY]} "$LOG_TAG" "$LOG_MESSAGE"
        done < <( journalctl --follow -o json --no-pager --no-tail | jq -r '"\(.SYSLOG_FACILITY // 3) \(.PRIORITY // 6 ) \(.SYSLOG_IDENTIFIER // "journald") \(.MESSAGE | sub("\\n";" ";"g") // "no message")"' )
}&

This code is part of a special initramfs. I have socat command available. I need to parse journal in realtime and send facility, priority, tag and message to different destination (log file, Plymouth console plugin, ...)

Any tips would be greatly appreciated. (This code will be part of systemimager software (GPL)) https://github.com/finley/SystemImager/wiki

Any tip would be greatly appreciated.

Olivier LAHAYE
  • 31

1 Answers1

0

I'm not sure what you gain by using <(...) instead of just piping:

journalctl ... |
jq -r ... |
while read LOG_FACILITY LOG_SEVERITY LOG_TAG LOG_MESSAGE
do logmessage ...
done

Of course, both versions have the same buffering problem.

However, in either case if you want to use socat to run the jq command you can do so by replacing jq -r ... by 

socat -u EXEC:'jq -r ...',pty,ctty STDIO

but there is a big problem with having to quote the jq arguments as socat will munge them a little and it is difficult getting the string through correctly. One way round this is to save the command in a shell variable and then use it later:

export cmd="$(cat <<\! 
jq -r '"\(.SYSLOG_FACILITY // 3) \(.PRIORITY // 6 ) \(.SYSLOG_IDENTIFIER // "journald") \(.MESSAGE | sub("\\n";" ";"g") // "no message")"'
!
)"

socat -u SYSTEM:'eval $cmd',pty,ctty STDIO

But finally, none of this should be necessary as jq takes an option --unbuffered which will achieve the desired effect.

meuh
  • 51,383
  • You're right, I discovered that the issue was jq lacking the --unbuffered and saw your detailed answer just after. This now works perfectly. As for the < <( use, I took the habit as the important part is the while and if it is behind the pipe, it is run in a sub process and variables are lost. Of course in this particular case, this Is not needed. – Olivier LAHAYE Sep 11 '19 at 13:37
  • 1
    The < <( aside avoiding the while to run in a sub process let me avoid to export the SEVERITY and FACILITY table as well. – Olivier LAHAYE Sep 13 '19 at 08:04