5

I'm new to Linux and trying to understand how redirections work.

I have been testing various syntaxes for redirecting stdout and stderr to the same file, which don't all produce the same results.

For example, if I try to list 2 files that don't exist (file1 and file2) and 2 that do (foo and fz):

Syntax #1 (without redirection):

$ ls file1 foo fz file2

Here's the output I get in terminal:

ls: cannot access file1: No such file or directory
ls: cannot access file2: No such file or directory
foo  fz

Syntax #2:

Now, with redirection:

$ ls file1 foo fz file2 > redirect 2>&1

The redirect file contains the same as the result for Syntax #1:

ls: cannot access file1: No such file or directory
ls: cannot access file2: No such file or directory
foo
fz

So with both of the above syntaxes, it seems that the shell prints stderr first, then stdout.

Syntax #3:

Now, if I try with either of the following syntaxes:

$ ls file1 foo fz file2 > redirect 2> redirect

or

$ ls file1 foo fz file2 2> redirect > redirect

Then the redirect file will contain this:

foo
fz
nnot access file1: No such file or directory
ls: cannot access file2: No such file or directory

Here it looks like stdout is printed before stderr, but then we see that the beginning of stderr is "cropped" by the same number of characters as stdout.

The stdout is 6 chars long (foo fz, carriage return included), so the first 6 chars of the stderr (ls: ca) have been overwritten by stdout. So it actually seems like stderr was printed first, and that stdout was then printed over stderr instead of being appended to it.

However, it would have made more sense to me if stderr had been completely erased and replaced with stdout, rather than just partially overwitten.

Syntax #4:

The only way I have found to correct Syntax #3 is by adding the append operator to the stdout:

$ ls file1 foo fz file2 >> redirect 2> redirect

or

$ ls file1 foo fz file2 2> redirect >> redirect

Which produces the same as Syntax #2:

ls: cannot access file1: No such file or directory
ls: cannot access file2: No such file or directory
foo
fz

This article here explains that Syntax #3 is wrong (presumably, so is Syntax #4). But for argument's sake: why is Syntax #3 wrong? What exactly is it telling (or not telling) the shell to do as opposed to Syntax #2?

Also, is there a reason why the output always displays stderr before stdout?

Thank you!

LePoufCelebre
  • 51
  • This doesn't answer your question, but the simplest way to redirect both stdout and stderr to the same file is &>, as in ls file1 foo fz file2 &> redirect. – gardenhead Mar 11 '16 at 04:15
  • 1
    @gardenhead Thanks, yes, I have come across this, but as I'm only a novice I'm really trying to break things down to make sure I understand everything correctly. Also I've read that while &> is a simpler and more modern way to redirect both, it won't work on older shells, and that it is safer to get in the habit of using > redirect 2>&1 for now. – LePoufCelebre Mar 11 '16 at 12:48

3 Answers3

6

It's just like running two processes to write to the same file at the same time...bad idea. You wind up with two different open file handles and your data can get garbled (as it does in #3 above). Using syntax #2 is correct; it makes one file handle and points both stderr and stdout to the same place.

As for stderr always being printed first, there is no rule on this whatsoever. I suspect with ls it is because ls needs to check every entry in the directory before it can actually state that a particular file doesn't exist. So rather than make N passes over the directory table, it makes a single pass, checking for all command line arguments given, reports the errors, and prints the files it found. Other commands may print to stderr after stdout, or even alternate between them.

Wildcard
  • 36,499
  • Right, thank you. I had a feeling the print order of stderr and stdout might be inherent to the command itself. So Syntax #3 is bound to make a mess then. Despite producing what looks like a clean output, Syntax #4 also uses two separate processes targetting the same file, so I assume it should be avoided as well. – LePoufCelebre Mar 11 '16 at 16:14
  • @LePoufCelebre, correct. Syntax #2 is the only correct way to point stderr and stdout to the same place. (Well, ls 2> redirect >&2 is fine also.) – Wildcard Mar 11 '16 at 16:47
6

Adding to wildcard's answer, stdout is usually buffered (stored in memory to be written out later), while stderr never is (you want error messages to show up, even if the program crashes before writing anything out). So stderr will usually show up earlier.

vonbrand
  • 18,253
2

Not a developer here or a kernel expert. So I can not comment on any code but in syntax #3 you are piping stdout and stderr into the same file using two different pipes. So the common file you are sending both outputs to, is possibly having resource contention issues from 2 different pipes. Considering non-real time nature of the OS, two pipes are feeding information into the same file, probably at the same time. Which might explain missing characters.

In syntax #2, you redirecting the stderr into the same pipe for stdout At that point, the file has one pipe coming in and shell manages this pipe against race conditions.

This is my educated guess and a guess only. Nothing to substantiate it.

MelBurslan
  • 6,966