7

I'm working on some script that being run by rc.local at startup, and I noticed that output redirection works quite strange.

If I write something like echo "foo" >&1, it ends up in syslog, and all is okay.

But when I write echo "foo" >>/dev/stdout, or echo "foo" >>/proc/self/fd/1, I got an error saying that there is no such device or address.

Further investigation revealed that /proc/self/fd/1 was in fact a socket. Including ls -l /proc/self/fd in rc.local prints the following:

lr-x------. 1 root root 64 Jul 14 05:28 0 -> /dev/null
lrwx------. 1 root root 64 Jul 14 05:28 1 -> socket:[18485]
lrwx------. 1 root root 64 Jul 14 05:28 2 -> socket:[18485]
lr-x------. 1 root root 64 Jul 14 05:28 255 -> /etc/rc.d/rc.local

The described behaviour can be easily reproduced with nc. First, find out which file descriptor binds to socket. You can do it by issuing nc -l -p 25566 -c "ls -l /proc/self/fd" in first terminal and then telnet localhost 25566 in other terminal. In my case it was 5-th descriptor.

Okay. Then, to reproduce the issue, in first terminal:

nc -l -p 25566 -c "echo 'hello' >&5"

in second terminal:

telnet localhost 25566

You shall see "hello" in second terminal output between telnet messages about connection being established and closed.

Now the case with file descriptor pseudo-file from /proc:

first terminal:

nc -l -p 25566 -c "echo 'hello' >/proc/self/fd/5"

second terminal:

telnet localhost 25566

Now second terminal just contains telnet messages about connection established and immediately closed, and first terminal shows an error: sh: /proc/self/fd/5: No such device or address.

Edit: OS is Fedora 23 i686 server


So, the question. What's the difference between >&1 and >/proc/self/fd/1 ? And is there some universal and reliable way to redirect output to some file, which corresponds exactly to current standard output?

Thank you.


Edit 2:

For clarity, exact input/output for Fedora 23 i686 for above case:

Terminal 1                                       | Terminal 2
$ nc -l -p 25566 -c 'ls -go /proc/self/fd'       | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
                                                 | Escape character is '^]'.
                                                 | total 0
                                                 | lr-x------ 1 64 Jul 14 08:56 0 -> pipe:[19687]
                                                 | l-wx------ 1 64 Jul 14 08:56 1 -> pipe:[19688]
                                                 | lrwx------ 1 64 Jul 14 08:56 2 -> /dev/tty2
                                                 | lr-x------ 1 64 Jul 14 08:56 3 -> pipe:[19687]
                                                 | lr-x------ 1 64 Jul 14 08:56 4 -> /proc/1285/fd
                                                 | lrwx------ 1 64 Jul 14 08:56 5 -> socket:[19686]
                                                 | l-wx------ 1 64 Jul 14 08:56 7 -> /pipe:[19688]
                                                 | Connection closed by foreign host.
$ nc -l -p 25566 -c "echo 'hi' >&5"              | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
                                                 | Escape character is '^]'.
                                                 | hi
                                                 | Connection closed by foreign host.
$ nc -l -p 25566 -c "echo 'hi' >/proc/self/fd/5" | 
                                                 | $ telnet localhost 25566
                                                 | Trying ::1...
                                                 | Connected to localhost.
sh: /proc/self/fd/5: No such device or address   | Escape character is '^]'.
                                                 | Connection closed by foreign host.
$                                                | $
  • What operating system? – ctrl-alt-delor Jul 14 '16 at 14:34
  • Fedora 23 i686. Sorry, forgot to mention. I updated the post. – alexey.e.egorov Jul 14 '16 at 14:36
  • 2
    A command directs to stdout by default, unless you redirect to somewhere else. – ctrl-alt-delor Jul 14 '16 at 14:44
  • When I do nc -l -p 25566 -c "ls -l /proc/self/fd" I get the socket attached to fd/0 and fd/1 this is what I expected. – ctrl-alt-delor Jul 14 '16 at 14:49
  • Can you show us EXACTLY what you are doing and what happens. I mean EXACTLY, type in what you did, then do it based on exactly what you typed, then paste the result. (I am detecting noise between what happened and what is appearing here.) – ctrl-alt-delor Jul 14 '16 at 14:52
  • @richard, sorry for that cryptic echo >&1 line, it's just I pinpointed the problem and thought that for clarity it would be nice to go academic. – alexey.e.egorov Jul 14 '16 at 15:51
  • You seem to have a lot of fds open, I just have 4 and they are what I would expect. 0 and 1 are connected to a socket, that is what nc does (connect stdin and stdout to a network socket). Yours seem to be connected to a pipe (that makes no sence). 2 is connected to a /dev/pts (terminal), well I did not redirect. Yours to /dev/tty (just a different spelling), 3 to /proc/xxxx/fd (the directory where this info is), for you this is on 4, but then you have a load of other stuff, pipes on stdin/stdout, and 2 more pipes, – ctrl-alt-delor Jul 14 '16 at 21:58
  • What happens if you do nc -l -p 25566 -c "echo 'hi' >/proc/self/fd/1", this is how mine is wired, and how nc should wire it, and how yours seems to be wired if you do not look to close (see your first command: nc -l -p 25566 -c 'ls -go /proc/self/fd' it is not redirected so it goes to fd1, but we see that it goes through the socket, as telnet prints the result. As expected), I am wondering if those two pipes looping back on them selfs and the socket in a strange location, is just readhats way to get it to work, and you are not supposed to look inside, just use it. – ctrl-alt-delor Jul 14 '16 at 22:05

2 Answers2

9

Use >&N. It's portable and as you saw, actually works with sockets.

The only reason you would ever use /proc/self/fd is you are running a program that expects a file name and can't be told to use an already open file descriptor. E.g. the <(cmd...) redirection uses that, since almost all command line utilities can open a file pointed to by name, but not all support preopened file descriptors directly.

Your shell can use pre-existing file descriptors, though, so no need to go through /proc.


Also, /proc/NNN/fd/ is specific to Linux, and requires that you have a mounted /proc. On my Linux boxes, /dev/stdout, /dev/fd/* and others are symlinks to /proc/self/fd/* etc., so they require /proc too. On other Unixes, they might be different. According to the answers to an older question /dev/stdout are specifically listed as outside POSIX.

As for why the redirection doesn't work as you tried: Trying out with strace, the difference between the two is that for a >&N redirection, bash calls dup() on the file descriptor, and for >/proc/self/fd/N it just tries to open it as an ordinary file with open(). Apparently proc doesn't support opening new copies of sockets like that, so the call fails. Stream sockets are pretty much point to point links, so prohibiting opening of a new copy doesn't seem too unnatural. But why it works for pipes or with a dup, I couldn't tell.

$ ls -l /proc/self/fd/3
lrwx------ 1 itvirta itvirta 64 Jul 14 18:24 /proc/self/fd/3 -> socket:[168157]
$ strace bash -c "echo foo >>/proc/self/fd/3" 2>&1 | grep open.*proc/self
open("/proc/self/fd/3", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 ENXIO (No such device or address)

Also this answer has some information about the portability of /proc/NNN/fd

ilkkachu
  • 138,973
  • Though doing echo >&1 literally is a bit silly, because stdout already goes to fd 1, so the redirection doesn't do anything. – ilkkachu Jul 14 '16 at 15:41
  • Oh! So that's dup() and open(). Thanks, that explains a lot. Also I should have thought of strace'ing bash myself. Eeh... Seems quite obvious now. – alexey.e.egorov Jul 14 '16 at 15:49
3

A redirection like 3>&1 duplicates an existing file descriptor: this takes the same open file (same file, same flags, same position, etc.) and plugs it onto another “output port” of the program (another file descriptor number). (More precisely, this creates a new file descriptor that points to the same file description, but we don't)

>&1 duplicates the descriptor onto itself, some shells optimize it away altogether.

A redirection like >/proc/$pid/fd/1 opens the file /proc/$pid/fd/1. This creates a fresh file descriptor. The files in /proc/*/fd are special, and opening them mostly copies the data from an existing file descriptor. Although the files are symbolic links, they are “magic”; for example, a deleted file or a pipe appears as a broken symlink but can still be opened as if it was an existing file because the kernel contains special code to handle /proc/*/fd entries. So in most cases, >&1 and >/proc/self/fd/1 are equivalent. However, sockets get different treatment.

/*
 *      In theory you can't get an open on this inode, but /proc provides
 *      a back door. Remember to keep it shut otherwise you'll let the
 *      creepy crawlies in.
 */

static int sock_no_open(struct inode *irrelevant, struct file *dontcare)
{
        return -ENXIO;
}

(The code has been reorganized in recent versions, but sockets still can't be opened even via /proc/PID/fd/NUM.)

The reason you can't open a socket like most other file types is that there's more information attached to a socket, you have to say how you open it. For example, opening a TCP socket allocates a source port. Although it might make sense in some contexts, the Linux kernel doesn't allow opening another process's sockets. In the case where the process is the same (/proc/self/fd/NUM, as opposed to /proc/OTHERPID/fd/NUM), it could convert the open call to the usual file descriptor duplication, but opening /proc/self/fd is an unusual thing to do in the first place, and redirection with sockets isn't normally done since it normally doesn't work, so the kernel has not been designed to support this sensible-but-useless exception.