1

I am working on a process to send data via a pipe from one server to another for processing.

Although this is not the exact command, it might look something like this:

tail -f logfile | grep "abc" | grep "def" | grep -v "ghi" | netcat -q 0 n.n.n.n 7777

I would like to wrap all those greps into a script and more importantly prepend the pipe to netcat with an identifier, so the command would look like this:

tail -f logfile | myscript.sh {id}

The script listening on the other end should receive:

{id}
[Line 1 of the logfile]
[Line 2 of the logfile]
...

Wrapping it in a script is easy:

#!/bin/sh
id=$1
grep "abc" | grep "def" | grep -v "ghi" | netcat -q 0 n.n.n.n 7777

but I cannot figure out how to inject $id at the start.

The receiving end is using

 socat -u tcp-l:7777,fork system:/dev/receivePipe

so if there is a different way I could get the id (for example somehow as a parameter to /dev/receivePipe), or via an environment variable, that would work too.

EDIT: The final answer was figured out in the comments of the accepted answer:

#!/bin/sh
{
  printf '%s\n' $1
  grep "abc" | grep "def" | grep -v "ghi" 
} | netcat -q 0 192.168.56.105 7777

1 Answers1

3

Just do:

#! /bin/sh -
{
  printf '%s\n' "${1-default-id}"
  awk '/abc/ && /def/ && ! /ghi/'
} | socat - tcp:n.n.n.n:7777

${1-default-id} expands to the first positional parameter if specified or default-id otherwise. Replace with ${1?} to exit with an error if not passed any argument instead (or ${1?The error message} to specify an error message instead of the default).

We redirect the output of a command group that runs printf to output the ID and the filtering commands (here with your grep pipeline replaced with a single awk invocation that does the same thing a bit less clumsily) to socat/netcat.

Or to only print the ID if and when one line has been read and matches:

#! /bin/sh -
ID=${1-default-id} awk '
  /abc/ && /def/ && ! /ghi/ {
    if (!already_printed++) print ENVIRON["ID"]
    print
  }' | socat - tcp:n.n.n.n:7777

Or to prepend the ID (and a space character) to every line:

#! /bin/sh -
ID=${1-default-id} awk '
  /abc/ && /def/ && ! /ghi/ {
    print ENVIRON["ID"], $0
  }' | socat - tcp:n.n.n.n:7777

Beware awk, like grep will buffer their output when it goes to a pipe (to anything other than a tty device). With the GNU implementation of awk (aka gawk), you can add a call to fflush() after each print to force the flushing of that buffer. See also the -Winteractive of mawk. In most awk implementations, doing a system("") would also force a flush. The GNU implementation of grep has a --line-buffered option to force a flush after each line of output.

Also note that tail -f logfile is short for tail -n 10 -f logfile. Chances are you actually want either tail -n +1 -f logfile for the whole log file to be processed, and then tail carrying on following the file, or tail -n 0 -f logfile to process only the lines being added from now on.

  • I am a little confused as to why you are piping to socat. Did you mean to pipe to netcat, or is something else going on? Regardless, I don't want to convert the pipes to other things such as awk. The greps were just an example for this post. In reality, it might be other intermediary programs that it's getting piped through, with no greps at all. All I want to do is inject a line at the start of what ends up going to netcat. – Ben Holness Aug 01 '22 at 12:39
  • @BenHolness, seeing that you already used socat, I replaced your netcat call with the socat equivalent. In any case, the first approach will allow you to insert the header before any command. – Stéphane Chazelas Aug 01 '22 at 13:48
  • I used netcat on the sending server as socat is not installed there. I'm trying to understand how to use the first approach without awk and with the other pipes. I tried removing the awk line entirely (didn't work) and taking out all of the /abc/ parts in the awk line (also didn't work). Here's an example of what I tried (the one without awk), with different, perhaps non-sensical, intermediary programs (and squashed because multi-line comments aren't possible):

    #!/bin/sh { printf '%s\n' $1 } | grep "abc" | head -n 10 | netcat -q 0 192.168.56.105 7777

    – Ben Holness Aug 01 '22 at 13:55
  • @BenHolness, replace the awk '/abc/ && /def/ && ! /ghi/' (which is the same as grep abc | grep def | grep -v ghi) with your filtering command (like grep abc | head or grep -m10 abc) – Stéphane Chazelas Aug 01 '22 at 14:10
  • Got it, with (for example): #!/bin/sh { printf '%s\n' $1 grep "e" | head -n 3 } | netcat -q 0 192.168.56.105 7777

    Thanks!

    – Ben Holness Aug 01 '22 at 14:15