8

I intend to pipe the output of a program into a while read VAR loop and break when a pattern is found, but it doesn't.

Proof of concept:

inotifywait -qm -e create . | while read line; do echo $line; break; done
./ CREATE newfile

..

tail -f /var/log/syslog | while read line; do echo $line; break; done
Nov 6 22:44:05 section9 ntpdate[2381]: adjust time server 91.189.89.199 offset 0.272779 sec

These never exit no matter what the source program outputs. Setting set -x beforehand suggests that the loop never iterates to the second read. $BASH_SUBSHELL is 1 in these examples.

Shouldn't tail, inotifywait, etc. receive SIGPIPE and exit?

Note that process substitution (while read ... break; done < <(tail -f ...)) works OK. $BASH_SUBSHELL is 0 in this case.

arielCo
  • 1,058

2 Answers2

14

The key is that, under bash (other may shells differ), a pipeline is not terminated until all commands in the pipeline are finished.

To understand, let's consider:

inotifywait -qm -e create . | while read line; do echo $line; break; done

When read reads a line, it is echoed and then break is executed and the last process terminates. The first process, however, continues until a write to stdout fails. Thus, the loop will continue at least until inotifywait tries to write its second line of output. Because of the vagaries of buffering, it may not happen even then. It may take several lines before the discovery occurs. When that attempted write fails, then SIGPIPE is issued.

Now, consider the other case:

while read line; do echo $line; break; done < <(inotifywait -qm -e create .)

Here, there is no pipeline. When break is executed, the while loop is done.

Documentation

From the "Pipelines" section of man bash:

The shell waits for all commands in the pipeline to terminate....

This behavior is a choice made by bash. It is not mandated by POSIX. POSIX states:

If the pipeline is not in the background (see Asynchronous Lists), the shell shall wait for the last command specified in the pipeline to complete, and may also wait for all commands to complete.

John1024
  • 74,655
  • 1
    "the loop will continue at least until inotifywait tries to write its second line of output." There's the rub! SIGPIPE isn't issued when the reader process ends, but when a write fails. Also, the pipe can hold a lot: tail manages to write some 30K without getting a broken pipe. Thanks! – arielCo Nov 09 '14 at 05:16
  • I suppose inotifywait exits because its output is flushed on every line (I don't see any calls to fflush, though). – arielCo Nov 09 '14 at 05:23
  • 1
    I was almost about to /wrists before coming across this answer, thought I was going crazy. Thanks OP! – SleepyCal Aug 11 '21 at 19:25
2
strace tail -f -n 5000 /var/log/messages | 
    while read line; do echo $line; break; done;

shows in detail what happens:

[...]
write(1, "Nov  1 22:20:01 inno systemd[1]:"..., 4096) = 4096

i.e. tail writes up to 4096 bytes at once to the pipeline (that is what a pipeline can buffer). Usually there is not so much data at all for tail:

strace tail -f /var/log/messages | while read line; do echo $line; break; done;

shows

write(1, "Nov  7 07:00:02 inno systemd[1]:"..., 730) = 730

Thus tail would not try to write again until something is appended to the file.

I thought unbuffer would solve this problem. I am surprised that it doesn't.

unbuffer tail -f file1 | while read line; do echo $line; break; done
Hauke Laging
  • 90,279
  • The pipe can hold quite a bit when buffered - tail manages to write some 30K without getting a broken pipe. As John1024 said in the other answer, what I was missing is that SIGPIPE is received when a write fails, not when the process at the other end exits. – arielCo Nov 09 '14 at 05:24
  • @arielCo Write on what level? To the kernel syscall? Not under Linux I guess. The standard libraries do their own buffering though. – Hauke Laging Nov 09 '14 at 05:28
  • yes, the calls to write() in Linux succeed even after the downstream/reader process is gone, until enough has been written to "jam" the pipe. For a few hundred bytes as in tail -f, the writer won't notice that anything went wrong on the other side of the pipe. Now in tail -5000f /usr/share/dict/american-english, write() fails with EPIPE(-1) after a few tens of KB written. And inotifywait is "unbuffered" so it fails trying to write the second line. – arielCo Nov 09 '14 at 05:48