17

I am trying to write a script which is POSIX compliant, so it can run on any *nix system (Debian, Fedora, CentOS, AIX, Arch... ALL of them). When it comes to redirection, I am rather confused on what is supported by POSIX and what is not.

If I redirect ls -l file missingfile &> out.txt this works perfectly in bash. The line from stderr complaining that missingfile doesn't exist, and the permissions output from file, are both in out.txt. However, I believe this is only working because these shells support more than just POSIX standard operators. Upon researching this question I've come across conflicting answers.

This stackexchange answer, for example, seems to imply that &>, >&, >>& and &>> are non-standard operations. However, this answer explicitly states that some-program > some_file 2>&1 is POSIX compliant. Does this mean that the >& operator (using digits for stderr & stdout) is POSIX compliant whereas using &> to auto redirect both stderr and stdout to a file is not POSIX compliant? Or are both &> and >& not POSIX compliant / POSIX compliant (one of these two people are wrong)?

I considered just avoiding the & sign completely and using ls -l file missingfile >out.txt 2>out.txt however this comes with its own issue. Running the ls command this way causes the shell to open two file handles for out.txt, both pointing to offset 0 in the file. So when ls looks for these two files, one of the messages gets clobbered.

What one would expect ls -l file missingfile >out.txt 2>out.txt to output:

-rw-r--r-- 1 blah blah 0 Jun  3 13:18 file
ls: cannot access 'missingfile': No such file or directory

What is actually output:

-rw-r--r-- 1 blah blah 0 Jun  3 13:18 file
le or directory

What is the best way to redirect both stdout and stderr to a file in a POSIX compliant fashion?

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
ExecutionByFork
  • 499
  • I updated the misleading part of https://unix.stackexchange.com/a/159514/229446 – Gilles 'SO- stop being evil' Jun 03 '20 at 19:07
  • 2
    &> out.txt doesn't work in Dash. It does in Busybox sh, though, along with Bash and Zsh. – ilkkachu Jun 03 '20 at 19:44
  • @ilkkachu You are correct. I got confused on which shell I was using when I was testing it. I've updated my question – ExecutionByFork Jun 03 '20 at 22:03
  • 4
    This isn't really about portability across operating systems, but portability across shells. One can get variance in what provides the sh command even simply among versions of the same operating system in some cases. – JdeBP Jun 04 '20 at 00:57
  • 1
    There is a shell called dash which is meant not to have any extension to POSIX standard. So if your script works in dash, it will work in bash, ksh, ash etc. – Lorinczy Zsigmond Jun 07 '20 at 05:28

1 Answers1

41

You can check the POSIX specification. The language isn't always easy to follow, but if you know what you're looking for, you can see whether it's there or not.

POSIX includes the redirection operations N>&word and N<&word where N is an optional file descriptor number (defaulting to 0 for input and 1 for output) and word can be:

  • A sequence of digits, to redirect file descriptor N to whatever is already open on word.
  • The character -, to close file descriptor N.

So >out.txt 2>&1 is a POSIX-compliant way to redirect both standard output and standard error to out.txt. It also works in pre-POSIX Bourne shells.

>& followed by a file name is an extension in some shells (including bash) that redirects both stdout and stderr to that file.

&> is a synonym of >& followed by a file name in bash (and zsh). It has the advantage of working even if the file name is a sequence of digits or - (for example foo >&"$a" is equivalent to foo 1>&3 2>&1 if the value of a is the digit 3, but to foo >myfile 2>&1 if the value of a is myfile). Its downside is that it is incompatible with POSIX. In a POSIX-compliant shell, echo foo &>bar is parsed as the command echo foo, the & operator, and the command >bar: a background process that prints foo, and a foreground process that creates or empties the file bar.

Gilles 'SO- stop being evil'
  • 829,060
  • 1
    Interestingly enough, the POSIX-compliant way (>file.txt 2>&1) also works for Windows' command line. However, instead of - to close the file descriptor, you redirect to nul (same as /dev/null). – Ismael Miguel Jun 04 '20 at 10:23
  • @IsmaelMiguel Watch out. Does redirection to NUL actually close the fd (thus return errors on I/O, like 2>-), or just discard writes/return EOF on reads (like 2>/dev/null)? – TooTea Jun 04 '20 at 14:01
  • 1
    @TooTea As I said, you use nul to redirect the output stream, not to close it. nul is the same as /dev/null. From what I've seen, there's no file descriptor closing. https://ss64.com/nt/syntax-redirection.html <-- should give you a better light on what I mean – Ismael Miguel Jun 04 '20 at 14:14
  • In the last paragraph, shouldn't it read &> filename? – rexkogitans Jun 04 '20 at 19:36
  • @rexkogitans No, it's about >&, but I see how it could be hard to follow. I've reworded the last two paragraphs. – Gilles 'SO- stop being evil' Jun 04 '20 at 19:47
  • I'm a bit confused about - - can I use >- in place of >/dev/null? – xeruf May 07 '21 at 10:15
  • 1
    @Xerus Sometimes. It isn't the same thing, but in some circumstances it has the same effect. >/dev/null redirects the file descriptor whereas >- closes it. If the application tries to write and ignores write failures, this has the same effect. If the application exits on write failures, this likely doesn't have the same effect. If the application opens a file and it lands on that descriptor, this could have a completely different effect, e.g. ending up writing log messages to the file intended for normal output. – Gilles 'SO- stop being evil' May 07 '21 at 10:18