5

Standard file descriptors <= 2 are opened by default. A program can write to or read from, a file descriptor after 2, without using the open system call to obtain such a descriptor. The program can then advertise in its manual, which file descriptors it is using and how. To make use of this, a POSIX shell can open a file and assign that file to a descriptor with the exec built-in. After that, the shell would start the program which will use that descriptor and file.

One reason for doing that would be if the program wants to have more than one output or input file, and does not want to specify them as command line arguments. If there was just one file, you could just redirect a standard file descriptor.

I have never seen a generally available program which would advertise such a thing in its manual. Does this happen in practice? Has anybody heard of such a thing?

Yes, I do want to stay within POSIX world - so no bash-only built-ins. I just want to know if there is such a program, not a shell built-in.

AdminBee
  • 22,803
  • 1
    The bash shell's read builtin has an option -u fd to read from file descriptor fd instead of standard input - does that count? – steeldriver Aug 10 '16 at 10:54
  • 3
    Did you look at UCSPI-TCP? It does exactly what you describe. But if you say it isn't relevant, I'll just delete my comment. – Satō Katsura Aug 10 '16 at 11:06
  • @SatoKatsura I did not mean to be rude... I am sorry if it seemed that way. You were talking about how open call returns descriptors. I know that - but I specifically said in the question, I don't want that situation. – Mark Galeck Aug 10 '16 at 11:10
  • @steeldriver good point, thank you - but yes, I would like POSIX only. I tried your command, yes it works under bash but not under dash. Let me update the question. – Mark Galeck Aug 10 '16 at 11:11
  • I'm not sure if there is any program using a FD more than 2 or not, but I thinks this can help you. –  Aug 10 '16 at 11:23
  • The confusion is because you wrote the question poorly, with "use" instead of "expect to have already open at program startup" or some such. "use" implies a lot wide range than that, but it's only that that you were asking about. – JdeBP Dec 17 '16 at 19:19

2 Answers2

7

When you use process substitution with <(...) or >(...), bash will open a pipe to the other program on an arbitrary high file descriptor (I think it used to count up from 10, but now it counts down from 63) and pass the name as /dev/fd/N on the command line of the first program. This isn't POSIX, but other shells also support it (it's a ksh88 feature).

That's not exactly a feature of the program you're running though, it just sees /dev/fd/N and tries to open it like a regular file.

  • The Autoconf manual mentions some historic notes:

    A few ancient systems reserved some file descriptors. By convention, file descriptor 3 was opened to /dev/tty when you logged into Eighth Edition (1985) through Tenth Edition Unix (1989). File descriptor 4 had a special use on the Stardent/Kubota Titan (circa 1990), though we don't now remember what it was. Both these systems are obsolete, so it's now safe to treat file descriptors 3 and 4 like any other file descriptors.

  • Also while I did a google search for this I found a program called runit that uses file descriptors 4 and 5 for some purpose related to log rotation.

  • And quoting from the svlogd man page:

    If svlogd is told to process recent log files, (...). svlogd also saves any output that the processor writes to file descriptor 5, and makes that output available on file descriptor 4 when running processor on the next log file rotation.

AdminBee
  • 22,803
Random832
  • 10,666
  • 1
    thank you for mention of runit svlogd that answers my question. Everything else that everybody wrote, unfortunately, does not understand the question. Not sure why, I really tried to make it really clear. – Mark Galeck Aug 11 '16 at 07:52
4

Yes; plenty.

Sato Katsura mentioned UCSPI, the Unix Client-Server Program Interface. Programs that adhere to this use standard input, standard output, and standard error as normal. But the toolset defines further standard open file descriptors beyond that. There are other toolsets that define similar standard file descriptors beyond the first three.

Some of them are even consistent with one another. Like stdaux and stdprn as standard file handles 3 and 4 from the PC-DOS/DR-DOS/MS-DOS world, there's actually a consistent standard usage of file descriptors 3, 4, 5, 6, and 7 in the worlds of some Unix and Linux terminal management, logging management, and UCSPI client softwares.

UCSPI client-side tools

6 and 7: standard fromserver and standard toserver

With UCSPI, people usually mainly remember the server-side protocol, where servers communicate with their standard inputs and outputs. On the less-remembered client end, however, tools such as tcpclient (from Bernstein's UCSPI-TCP), sslclient (from Baxter's/Gifford's/Hoffman's UCSPI-SSL), and tcp-socket-connect (from the nosh toolset) use further standard file descriptors. The client programs, that they all chain to, are defined as expecting open file descriptor 6 as reading from the network and open file descriptor 7 as writing to the network.

Bernstein's original ucspi-tcp toolset came with a bunch of nonce TCP tools build around tcpclient. For example: Here is Bernstein's simple http@ tool. (Felix von Leitner published a more complex one a decade and a bit ago.)

#!/bin/sh
echo "GET /${2-} HTTP/1.0
Host: ${1-0}:${3-80}
" | tcpclient -RHl0 -- "${1-0}" "${3-80}" sh -c '
  addcr >&7
  exec delcr <&6
' | awk '/^$/ { body=1; next } { if (body) print }'

Further reading

terminal management tools

3, 4, and 5: standard ctty, standard PTY master and slave

Daniel J. Bernstein's original PTY toolset incorporated extra standard file descriptor definitions into the toolset. File descriptor 3 was the controlling terminal (a convention that Bernstein suggested in his accompanying papers on job control in shells), file descriptor 4 was the master side of a PTY, and file descriptor 5 was its slave side.

The PTY toolset has perhaps the most extreme example of tools that define specific open file descriptor conventions in their manuals. ptyspawn's manual page discusses file descriptors 0, 1, 2, 3, 4, 5, and 9.

The nosh toolset's tools for terminal management share some of these. pty-get-tty, for example, passes the master side of the pseudo-terminal that it opens as file descriptor 4; and pty-run and console-terminal-emulator both expect to receive it there.

Further reading

LISTEN_FDS protocol

3 and up: standard listen

What UCSPI is to passing connected sockets, this protocol is to passing listening sockets. (It's in the name.) The original inetd protocol for listening sockets was that they were the standard input of the server process, and this is what Bercot's s6-networking toolset still does, for one. But in the systemd world, the idea was for server processes to listen on more than one socket simultaneously. Thus arose this protocol, which defines file descriptors 3 and upwards as being a sequence of N open file descriptors, where N is the value of the LISTEN_FDS environment variable.

So whilst it may not be explicit in their manuals, every server program that incorporates this protocol has a set of "standard listen" file descriptors that it expects.

The manual pages for tools such as tcp-socket-listen, which both take and add listening socket file descriptors, do explicitly mention these file descriptors, though. They also mention the lesser-known UPSTART_FDS protocol from the upstart world.

Further reading

logging tools

4 and 5: standard rotation data

As you've already observed, a convention the started with Bernstein's multilog was the convention for tools spawned by multilog and its successors at log rotation time. All of these define file descriptors 4 and 5 as a way of passing data from run to run of the rotation filter program.

Further reading

  • Jonathan de Boyne Pollard (2015). "Logging". The daemontools family. Frequently Given Answers.

user management tools

3: standard credentials

Bernstein's checkpassword interface, originally used for validating POP3 server login, defined another suite of tools expecting an extra file descriptor to be open. Once again, it is file descriptor 3, on which the spawning program is expected to have written, or to be writing, the user account credentials which the password check program validates. There's a whole subfamily of checkpassword-compatible programs nowadays.

Further reading

JdeBP
  • 68,745