1
{
echo bla1
echo bla2 1>&2
} >>myfile 2>&1

Is there any difference between the two echo-s?

The only difference I can think of, if echo bla2 2>&1 retains the unbuffered property from stderr.

How about

{
echo bla1
echo bla2 1>&2 &
} >>myfile 2>&1

?

Is it the same as

{
echo bla1
echo bla2 &
} >>myfile 2>&1

?

After {...} >>myfile 2>&1 will fd1 and fd2 essentially become the same thing, or do they retain anything from their past?

EDIT:

Building on the answer from @ilkkachu, I tried the following:

perl -e 'print "foo"; sleep 2; print "bar\n"; sleep 2; print "bla"'
# ^^^line buffered
perl -e 'print "foo"; sleep 2; print "bar\n"; sleep 2; print "bla"' | cat
# ^^^block buffered
touch myfile; tail -f myfile &
perl -e 'print "foo"; sleep 2; print "bar\n"; sleep 2; print "bla"' >>myfile 2>&1
# ^^^block buffered
perl -e 'print STDERR "foo"; sleep 2; print STDERR "bar\n"; sleep 2; print STDERR "bla"'
# ^^^unbuffered
perl -e 'print STDERR "foo"; sleep 2; print STDERR "bar\n"; sleep 2; print STDERR "bla"' 2>&1 | cat
# ^^^unbuffered
perl -e 'print STDERR "foo"; sleep 2; print STDERR "bar\n"; sleep 2; print STDERR "bla"' >>myfile 2>&1
# ^^^unbuffered

In case of fd1, the buffering is adjusted according to the output type.

In case of fd2, the output was irrelevant.

{ echo -n foo; sleep 2; echo bar; sleep 2; echo -n bla; }
# ^^^unbuffered
{ echo -n foo; sleep 2; echo bar; sleep 2; echo -n bla; } | cat
# ^^^unbuffered
touch myfile; tail -f myfile &
{ echo -n foo; sleep 2; echo bar; sleep 2; echo -n bla; } >>myfile 2>&1
# ^^^unbuffered
{ echo -n foo 1>&2; sleep 2; echo bar 1>&2; sleep 2; echo -n bla 1>&2; }
# ^^^unbuffered
{ echo -n foo 1>&2; sleep 2; echo bar 1>&2; sleep 2; echo -n bla 1>&2; } | cat
# ^^^unbuffered
{ echo -n foo 1>&2; sleep 2; echo bar 1>&2; sleep 2; echo -n bla 1>&2; } >>myfile 2>&1
# ^^^unbuffered

It seems, echo always flushes, regardless where the output is going.

The experiment was repeated with

{ printf foo; sleep 2; printf 'bar\n'; sleep 2; printf bla; }

as well, the results were identical to echo.

Result: In the above cases, the buffering was determined by the writer.

Zoltan K.
  • 493
  • Ah, ok, I meant echo as some program writing in stderr and stdout. – Zoltan K. Sep 15 '21 at 17:03
  • Do I understand this correctly: it is not "fd2", what is unbuffered, rather the stderr object the program is writing into? (if the writer program has no buffering of its own) – Zoltan K. Sep 15 '21 at 17:06
  • Ops, now I realized what is the problem with the echo, corrected to 1>&2. – Zoltan K. Sep 15 '21 at 22:32

1 Answers1

2

The output buffering that usually comes up is a property of the C runtime library, and has nothing to do with any redirections, or the shell.

When the process starts, it just gets some file descriptors. The runtime library then builds the internal structures with their associated buffering (if any) on top of those. All it cares there is if fd 1 (for stdout) is connected to a terminal or not: if it is, it's made line buffered, if not, it's block buffered. Stderr is made unbuffered anyway. Mostly it's about performance, system calls to write the output are relatively expensive, and if the output goes to a file or a pipeline, the assumption is that bandwidth matters more the latency.

Of course the buffering behaviour may depend on the utility, too, and some tools might well reimplement the C library behaviour by themselves. (I'm thinking things like Perl and Python might do that, but as far as I've seen, they follow the custom anyway.) Some utilities also have options to control the buffering, for the cases where the assumptions don't hold (e.g. --line-buffered with at least GNU and FreeBSD grep), and there are tools to force/cheat that on tools that can't do it themselves, see: Turn off buffering in pipe

Anyway, the output from echo is likely to be never buffered, for the simple reason that if it were an external command, it would have to flush it's buffers before returning. Though a shell that handles echo itself could buffer it (*).

To use Perl as a testing tool, this will print all the output at once after a delay (block buffering):

$ perl -e 'print "foo\n"; sleep 2; print "bar\n";' | sed -e 's/^/:/'
[delay...]
:foo
:bar

This will print the lines immediately, with a sleep in between:

$ perl -e 'print STDERR "foo\n"; sleep 2; print STDERR "bar\n";' 2>&1 | sed -e 's/^/:/'
:foo
[delay...]
:bar

(The point of the sed there is to verify the output goes through the pipe and not past it to the terminal.)

You could do the same with a simple C program.

In your question, I'm not sure what the redirection in echo bla2 2>&1 is supposed to do, given that echo prints to stdout, fd 1, so it doesn't touch the redirected file descriptor. If we do it the other way around, somecommand 1>&2, then the command will still be writing to its own stdout, with the associated buffering.

Like here (with line buffering this time):

$ perl -e 'print "foo"; sleep 2; print "bar\n";'  1>&2
[delay...]
foobar

Of course echo isn't the only one to flush buffers on exit, you'd get the same with any other tool if they exit before otherwise triggering buffers to flush. E.g. this would print in two parts, even though both perls would use line buffering:

$ perl -e 'print "foo"'; sleep 2; perl -e 'print "bar\n";'
foo[delay...]bar

(* and ksh seems to buffer in some cases. Basically it optimizes things like echo foo; echo bar; to just one write() call, unless there's pretty much anything else in between. Except on the version I have, a delay loop will show the behaviour. Something like ksh -c 'echo foo; i=0; while [ "$i" -lt 234567 ]; do i=$((i + 1)); done; echo bar' gives a delay first, and then both foo and bar, but this is a bit of a corner case.)

ilkkachu
  • 138,973
  • I have added an EDIT section to my answer. I think my results are consistent with what you wrote, could you confirm it? – Zoltan K. Sep 15 '21 at 23:16
  • 1
    @ZoltanK., looks sensible to me. Though note that it's not just echo that always flushes. You could similarly use e.g. perl -e 'print "foo"'; sleep 2; perl -e 'print "bar\n"' which should show no buffering. echo is a command, just like any other (perl, ls, sleep), and even if the shell implements echo as a builtin, in the general case the next command up might rely on the output being visible before it runs, so implementing the buffering there is of little use. – ilkkachu Sep 15 '21 at 23:49
  • 2
    +1. note also that some programs buffer output by default but also allow you to override output buffering. e.g. GNU grep's --line-buffered option. or perl's $| variable (see man perlvar). Options like these trade some performance (it's faster to output data in large blocks than line-by-line, which is especially useful when piping into another program) in exchange for more "responsiveness" in interactive use. – cas Sep 16 '21 at 10:15