What were the original reason(s) for adding a "standard error" which is separate from "standard output"?

Question

Unix provides standard output and standard error, which can be redirected independently.

$ ls /not-exist
ls: cannot access '/not-exist': No such file or directory

$ ls /not-exist > redirect.out
ls: cannot access '/not-exist': No such file or directory

$ ls /not-exist 2> redirect.err
$

I heard there's a story somewhere on the Web, which gives a fun reason why this separation was implemented. It involves the computerized typesetting that early Unix was used for (and Unix pipelines, I think). I failed to find it right now.

Would anyone like to link that story here, to associate it with the relevant tags and make it easier to find?

Answer 1

There's Steve C. Johnson's 2013 account of this, as a user, where users complain about phototypesetting and — lo! — the problem is fixed two days later. But Douglas McIlroy told the tale slightly differently a quarter of a century earlier.

In McIlroy's version, standard error was a natural consequence of Ken Thompson's famous all-nighter introduction of the Unix command pipeline. In the world of Unix prior to the pipeline, the fact that errors would be sent to the file where standard output had been redirected to was "trouble". But after the introduction of the pipeline, this behaviour "became intolerable when the output was sent to an unsuspecting process". McIlroy recounts that Dennis Ritchie introduced the standard error mechanism to finally rectify this "shortly" after Sixth Edition.

Also, McIlroy had of course been working on the idea of pipelines in Unix for a fair while, by this point, including a number of proposals over the period of at least 2 years; having invented the garden hosepipe metaphor half a decade earlier than that. The concept of a separate stream distinct from the pipeline streams did not magically appear from nothing in just a couple of days.

Further reading

• Steve C. Johnson (2013-12-11). Graphic Systems C/A/T phototypesetter. TUHS mailing list. The Unix Heritage Society.
• M Douglas McIlroy (1987). A Research UNIX Reader: Annotated Excerpts from the Programmer’s Manual, 1971–1986. AT&T Bell Laboratories Computing Science Technical Report #139. p. 9. (archive)
• "'I'm going to do it,' and so he did". The Creation of the UNIX Operating System.