3

I want to monitor a file. So I have tailed a file using below command, to execute a script per line.

tail -3f logfile.log | grep "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash testscript.sh

But it seems that script is not getting executed. I observed that grep is not giving any input for next pipe.

When I tried,

tail -3f logfile.log | grep "action.*add"

it worked. But when given next filter like sed, grep, xargs etc. It didn't worke like the one shown below.

tail -3f /var/tmp/rabbitmq-tracing/logfile.log | grep "action.*add" | grep add

Please help me to know why this is happening and how to overcome this.

Edit 1: Basically anything like below should work and it was working previously. Confused why it is not working now.

tail -f file.txt | grep something | grep something | grep something

EDIT 2: Output line after first grep will be a json string like below. And I want to give this line as input( enclosed in single quotes) to bash script.

{"twNotif": {"originator": "api", "chain": "test", "txId": "08640-0050568a5514", "version": "1.0", "msgType": "api", "twData": {"api": {"hostId": "007bdcc5", "user": "test", "cmdTxt": "100599"}}, "action": "add", "store": "test", "msgTime": 1467280648.971042}}
ilkkachu
  • 138,973
AVJ
  • 505

3 Answers3

10

use --line-buffered switch on grep

tail -3f logfile.log | grep --line-buffered "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash testscript.sh

from man grep:

--line-buffered Use line buffering on output. This can cause a performance penalty.

or you can use stdbuf read more

stdbuf allows one to modify the buffering operations of the three standard I/O streams associated with a program. Synopsis:

use this syntax:

... | stdbuf -oL grep ... | ...

your example:

tail -3f logfile.log | stdbuf -oL grep "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash testscript.sh
Baba
  • 3,279
  • if I use --line-buffered for grep then sed receives input. But sed do not give any input to next piped command. In such case, do sed have any such option like '--line-buffered'. – AVJ Jul 01 '16 at 12:34
  • 1
    @AVJ , no no! you use -u in sed; -u in sed is equal with --line-buffered in grep !! – Baba Jul 01 '16 at 13:22
  • Yeah it worked with -u in sed... But can u just explain why this happened. Because.. for me previously something like below command was working fine. tail -f logfile.log | grep pattern | grep pattern | grep pattern | sed 's/ss/gg/g' – AVJ Jul 01 '16 at 15:19
  • @AVJ : what is your testscript.bash code?! ,, this this on you pc http://pastebin.com/ix7zyGRY ,, you can see your sed have not problem; – Baba Jul 01 '16 at 17:35
4

As others have noted, grep's line-buffering is the obvious cause of your problem.

However, there are other problems with what you're doing that are not so obvious.

To start with, it looks like you're using sed to add apostrophes to the start and end of each output line so you can feed it into xargs. This isn't even remotely necessary - xargs has -d option you can use to tell it to use newlines as delimiter: e.g. xargs -d'\n' -r (the -r option is to make sure xargs does nothing if there are no input lines)

Secondly, you're using regular expressions to parse json data. This is unreliable, extremely difficult or even impossible to get right with complex/nested structures, fragile, and prone to break at the drop of a hat for exactly the same reasons that using regexps to parse XML or HTML is unreliable, extremely difficult, and fragile. Don't parse XML or HTML with regular expressions. It doesn't work. The same applies to json.

Instead, you should use something like jq or jsonpipe to extract fields from json data. For example:

jq -c 'select(.twNotif.action == "add")' file.txt | 
  xargs -d'\n' -r -L 1 -P 5 ./testscript.sh

If you only wanted to pipe the value of the action field to xargs (without the double-quotes), you could do it like this:

jq 'select(.twNotif.action == "add") | .twNotif.action' file.txt | 
  sed -e 's/"//g' | 
  xargs -d'\n' -r -L 1 -P 5 ./testscript.sh

Using jsonpipe and awk would probably be easier for a job like that:

jsonpipe < file.txt | 
  awk '$1 == "/twNotif/action" {gsub(/"/,""); print $2}' |
  xargs -d'\n' -r -L 1 -P 5 ./testscript.sh

(note the redirection there, unlike jq, jsonpipe works only with stdin).

BTW, jsonpipe transforms json data into a line-oriented format suitable for using with line-oriented tools like sed, grep, awk, etc. For example:

$ jsonpipe < file.txt
/   {}
/twNotif    {}
/twNotif/originator "api"
/twNotif/chain  "test"
/twNotif/txId   "08640-0050568a5514"
/twNotif/version    "1.0"
/twNotif/msgType    "api"
/twNotif/twData {}
/twNotif/twData/api {}
/twNotif/twData/api/hostId  "007bdcc5"
/twNotif/twData/api/user    "test"
/twNotif/twData/api/cmdTxt  "100599"
/twNotif/action "add"
/twNotif/store  "test"
/twNotif/msgTime    1467280648.971042

It's easier to work with than jq, especially if you don't need json-formatted output.

It's also useful for listing the structure of and fields/elements in a json file in form that can easily be used with jq. e.g.:

$ jsonpipe <file.txt | awk '{gsub(/\//,"."); print $1}'
.
.twNotif
.twNotif.originator
.twNotif.chain
.twNotif.txId
.twNotif.version
.twNotif.msgType
.twNotif.twData
.twNotif.twData.api
.twNotif.twData.api.hostId
.twNotif.twData.api.user
.twNotif.twData.api.cmdTxt
.twNotif.action
.twNotif.store
.twNotif.msgTime
Community
  • 1
cas
  • 78,579
1

The problem you're likely seeing is "pipeline buffering"; the output of various components of the pipeline (in your case the grep) are no longer line buffered but block buffered, and a buffer may be 4K or so The next component doesn't see data immediately. It will see it when enough data occurs...

The solution is a little painful. Fortunately the expect package comes with a comamnd unbuffer which can help. man unbuffer for more details on how to use this.

It works by making components of the pipeline think they are talking to a terminal and so stay line buffered.

Stephen Harris
  • 44,540