1

I have a script that reads a file that contains urls and then prints out the ip addresses. However, I am unable to find resources as to why the pipe blocks when I add the command 

cut -d" " -f4

at the end of the pipe. Here is the full command. 

for url in $(cat urls); do host $url; done | grep "has address" | cut -d" " -f4

The output is printed asynchronously till the grep command. Any command that comes after the grep has its output blocked till all the urls have been processed by the host command. Can anyone give me insight as to why this happens?

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
MykelXIII
  • 419
  • 2
    Turn off buffering in pipe – Stéphane Chazelas Jun 01 '16 at 11:24
  • See also https://www.google.co.uk/search?q=site:unix.stackexchange.com+"--line-buffered" – Stéphane Chazelas Jun 01 '16 at 11:27
  • @StéphaneChazelas I have realized that it might be a buffering issue. What I don't understand is why are some commands buffered while other are not? – MykelXIII Jun 01 '16 at 11:38
  • 1
    Every command does as it pleases, but generally, output is not-buffered or line buffered where it goes to a terminal, and block buffered (with buffers of varying size, generally at least 4KiB) when not (like for grep when its output goes to a pipe above). – Stéphane Chazelas Jun 01 '16 at 11:41
  • @StéphaneChazelas Thank you for your reply. I am still confused why the grep above buffers but I have used grep a lot of times and most of the time it does not buffer. I am trying to figure out the exact condition that caused the command above to buffer. Thank you. – MykelXIII Jun 01 '16 at 11:51

1 Answers1

1

grep buffers output if it does not write to a terminal. You can use grep --line-buffered to output each line immediately. Consider:

run slow output command and grep to terminal:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep to
printing
to grep
printing
to grep
printing
to grep

run slow output command and grep to cut:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep to | cut -d" " -f1-2
printing
printing
printing
to grep
to grep
to grep

run slow output command and grep to cut using --line-buffered option:

$ for i in {1..3}; do
>   echo printing >&2
>   echo to grep
>   sleep 1
> done | grep --line-buffered to | cut -d" " -f1-2
printing
to grep
printing
to grep
printing
to grep
adonis
  • 1,724