5

I use golang for writing programs that are too complex to express as bash functions. The programs read stdin line by line and write modified output to stdout.

The problem with this iterating over all the lines in stdin approach, is that it doesn't exit early. For example, the following command will process all the lines instead of just 10:

cat largefile | custom-program | head -n 10

How do the native unix application manage to exit early and how can I do the same in my custom scripts?

Kshitiz Sharma
  • 8,785
  • 2
    Normally, when a program writes to a closed pipe it receives a signal, SIGPIPE. This is not usually trapped and by default the program is killed. – meuh Nov 13 '17 at 14:47
  • @meuh yes, but that's only relevant if the command outputs the lines as it processes them, rather than waiting for all items to be processed and then printing them (which I suspect is his case). – jordanm Nov 13 '17 at 14:57
  • 1
    @meuh Perhaps this question would have made more sense on Stackoverflow. My followup question would be how is this SIGPIPE handled in golang or scala in which I'm writing my scripts? – Kshitiz Sharma Nov 13 '17 at 15:06
  • FYI your program might be broken. Or you may see some issues because of buffering. Don't have now time but I had hard time making journald -f exit after some log line is observed. It does exit but only after more log lines come and it tries to write again. – akostadinov Nov 13 '17 at 17:49

1 Answers1

14

In yes | head, yes is terminated by a SIGPIPE signal the next time it writes a line after the other end of the pipe is closed (when head exits).

In:

(seq 20; sleep 10; seq 10) | head

The subshell and children will be able to write the first 20 lines to the pipe (as that fits in the pipe buffer), and will then sleep 10 seconds unharmed. Meanwhile, head will have died, but it's only when the subshell writes more lines to the pipe that it will be killed.

If you wanted your command to be killed as soon as head returns, as opposed to the first time it writes something to the pipe after head has returned, with bash you could do:

{ head; kill "$!"; } < <(exec custom-program < large-file)

(bash so happens to store the pid of process substitution in $!).

See also Grep slow to exit after finding match? for a more portable approach, or limit find output AND avoid signal 13 for one applied to find.

Or for the reverse this answer (run_under_watch function) to a similar Q&A to kill a process as soon as its stdin becomes a broken pipe.

Stéphane Chazelas
  • 544,893