How to grep netcat output

Question

I'm trying to grep live text stream from netcat, but it doesn't work for me:

netcat localhost 9090 | grep sender

returns nothing, but I'm sure that it should.

If I redirect the netcat output to a file and add some delays (simulate real environment) - then it works:

$ (sleep 5; cat netcat_output; sleep 5) | grep sender

{"jsonrpc":"2.0","method":"GUI.OnScreensaverDeactivated","params":{"data": "shuttingdown":false},"sender":"xbmc"}}

I also tried to add --line-buffered but w/o success.

What I do wrong?

Edit:

I noticed the same issue with sed, the output is empty.

But, for instance, hexdump converts text to hex live:

$ netcat localhost 9090 | hexdump -C
00000000  7b 22 6a 73 6f 6e 72 70  63 22 3a 22 32 2e 30 22  |{"jsonrpc":"2.0"|
00000010  2c 22 6d 65 74 68 6f 64  22 3a 22 50 6c 61 79 65  |,"method":"Playe|
00000020  72 2e 4f 6e 50 6c 61 79  22 2c 22 70 61 72 61 6d  |r.OnPlay","param|
00000030  73 22 3a 7b 22 64 61 74  61 22 3a 7b 22 69 74 65  |s":{"data":{"ite|
00000040  6d 22 3a 7b 22 69 64 22  3a 36 2c 22 74 79 70 65  |m":{"id":6,"type|
00000050  22 3a 22 6d 6f 76 69 65  22 7d 2c 22 70 6c 61 79  |":"movie"},"play|
00000060  65 72 22 3a 7b 22 70 6c  61 79 65 72 69 64 22 3a  |er":{"playerid":|
00000070  31 2c 22 73 70 65 65 64  22 3a 31 7d 7d 2c 22 73  |1,"speed":1}},"s|

Maybe it gets buffered in the pipe? You could try using stdbuf -o0 netcat localhost 9090 | grep sender (taken from here) — user43791, Dec 29 '14 at 23:12
Does it work like this ? netcat -z localhost 9090 | grep sender — Gilles Quénot, Dec 29 '14 at 23:21
@user43791 the output is still empty, but I'm sure that netcat returned matching string — tomekceszke, Dec 30 '14 at 10:39
@sputnick this one immediately returns to shell and doesn't wait for an event — tomekceszke, Dec 30 '14 at 10:40
There are no new lines in the hexdump, so grep is probably waiting endlessly for a newline. cat works since grep will get an EOF if not a newline, at least. Maybe try awk with { as RS? — muru, Dec 30 '14 at 11:10

score 11 · Answer 1 · edited Apr 13 '17 at 12:36

11

Answer is here: grep not matching in nc output

netcat outputs verbose logs to standard error, so we need to capture errors before we pipe to grep.

$ netcat -zv localhost 1-9000 2>&1 | grep succeeded

edited Apr 13 '17 at 12:36

Community

1

answered Apr 02 '16 at 21:01

tkdave

211

This answer just made me solution. That was very strange when I run that command without 2>&1, it look like grep didn't work at all. – Marecky Jan 24 '18 at 22:31

score 4 · Accepted Answer · answered Dec 30 '14 at 21:58

You could use the read command (bash builtin) to force characters to be read one by one :

netcat localhost 9090 | (
    cnt=0
    line=
    while read -N 1 c; do
        line="$line$c"
        if [ "$c" = "{" ]; then
            cnt=$((cnt+1))
        elif [ "$c" = "}" ]; then
            cnt=$((cnt-1))
            if [ $cnt -eq 0 ]; then
                printf "%s\n" "$line"
                line=
            fi
        fi
    done
) | grep sender

This script should print every full output with balancing {and }, but you can change the script to do whatever you want. This script would NOT do well on a benchmark compared to pretty much anything, but it's pretty simple and seems to work for me...

Note that your test sample didn't have matching {and }, so if this is the case of the real input, you might want to change the criteria to print the line.

Looks like the mismatching braces is just because he only posted the beginning of the output. — Barmar, Dec 31 '14 at 19:09

muru · Answer 3 · 2014-12-30T11:51:47.157

2

I think the issue is the absence of newlines in the netcat output. I can see two workarounds:

Insert a newline every x seconds (with unfortunate consequences if the newline is inserted in the middle of source):
```
( while sleep 1; do echo; done & netcat ... ) | grep source
```

Use awk with an RS other than newline:

netcat ... | awk -v RS='}' -v ORS='}' '/source/'

edited Dec 30 '14 at 11:51

answered Dec 30 '14 at 11:42

muru

72,889

Still empty output. To make sure that command is correct I run on dumped stream - cat netcat_output | awk -v RS='}' -v ORS='}' '/sender/' The output was ok: ,"sender":"xbmc"} – tomekceszke Dec 30 '14 at 12:55
I just checked first workaround and it works! But as you mentioned it can split the output so it cannot be the final solution. Anyway many thanks! – tomekceszke Dec 30 '14 at 13:56

score 1 · Answer 4 · answered Dec 30 '14 at 07:03

1

use watch command:

watch 'netcat localhost 9090 | grep sender'

answered Dec 30 '14 at 07:03

αғsнιη

41,407

It doesn't work too. The same with sed... But, for instance, this: netcat localhost 9090 | hexdump works and replace text to hex live, why? – tomekceszke Dec 30 '14 at 10:32

score 0 · Answer 5 · edited Apr 13 '17 at 12:36

0

As per @user43791 comment, try disabling buffering in pipe, e.g.:

stdbuf -o0 netcat localhost 9090 | grep sender

edited Apr 13 '17 at 12:36

Community

1

answered Dec 29 '15 at 21:02

kenorb

20,988

score 0 · Answer 6 · answered Feb 11 '16 at 11:32

In Bash you can use pseudo-device files to open a TCP connection and grep it as usual. For example:

$ grep "sender" < /dev/tcp/example.com/1234

To test it, simply run server which can send some file, like:

$ nc -vl 1234 < /etc/hosts

Then in another terminal run the test to grep the output:

$ grep "127" < /dev/tcp/localhost/1234
127.0.0.1   localhost

How to grep netcat output

6 Answers6