Reading from file descriptor fails

Question

This question is about reading and writing on a file descriptor. See the following example:

#!/bin/sh

file='somefile'

# open fd 3 rw
exec 3<> "$file"

# write something to fd 3
printf "%s\n%s\n" "foo" "bar" >&3

# reading from fd 3 works
cat "/proc/$$/fd/3"

# only works if the printf line is removed
cat <&3

exit 0

This script outputs:

foo
bar

Expected output:

foo
bar
foo
bar

Opening and writing to the file descriptor succeeds. So does reading via proc/$$/fd/3. But this is not portable. cat <&3 doesn't output anything. However, it works when the file descriptor is not being written to (e.g. uncomment the printf line).

Why doesn't cat <&3 work and how to read the entire contents from a file descriptor portably (POSIX shell)?

Answer 1

cat <&3 does exactly what it's supposed to do, namely read from the file until it reaches the end of the file. When you call it, the file position on the file descriptor is where you last left it, namely, at the end of the file. (There's a single file position, not separate ones for reading and for writing.)

cat /proc/$$/fd/3 doesn't do the same thing as cat <&3: it opens the same file on a different descriptor. Since each file descriptor has its own position, and the position is set to 0 when opening a file for reading, this command prints the whole file and doesn't affect the script.

If you want to read back what you wrote, you need to either reopen the file or rewind the file descriptor (i.e. set its position to 0). There's no built-in way to do either in a POSIX shell nor in most sh implementations (there is one in ksh93). There is only one utility that can seek: dd, but it can only seek forward. (There are other utilities that may skip forward but that doesn't help.)

I think the only portable solution is to remember the file name and open it as many times as necessary. Note that if the file isn't a regular file, you might not be able to seek backwards anyway.

Answer 2

#!/bin/sh

exec 3>file
exec 4<file

printf "%s\n" "foo" "bar" >&3
cat <&4

By using separate file descriptors for reading and writing you get separate positions in the file. Writing doesn't change the reading position.

Answer 3

With ksh93, it is possible to seek:

#!/usr/bin/env ksh93
file='somefile'
exec 3<> "$file"
printf "%s\n%s\n" "foo" "bar" >&3
cat "/proc/$$/fd/3"
exec 3>#((0))
cat <&3
exit 0

I get:

foo
bar
foo
bar

as wanted.

Answer 4

You need lseek function to reposition file offset. It is not implemented in bash, so you need to write a basic program in C for that. An example of it, and some discussion can be found on bash mailing list:

int main(int argc, char * argv[])
{
    return lseek(atoi(argv[1]), 0L, 0);
}

Of course in such simple scenario as in question you can always just reinitialize descriptor

exec 3<> file

after printf, but that is obviously not general solution.

Answer 5

If you are doing some work on a file descriptor that you expect you'll want to read again, then you can portably do it in the body of a here-document:

exec 4<somefile 3<<plus
$(    printf %s\\n some random lines
        cat <&4
)
plus
cat <&3

The solution is not perfect perhaps - there is no portable way, for example, to seek back through the here document file descriptor. And the command substitution will elide trailing blank lines from whatever cat writes to stdout, but all of these problems are very simply handled.

In the first place, redirections are scoped to their containing compound command, and so it is a small matter to nest these a bit in a loop to handle any repetition you might require. It is also possible to implement a loop within the command sub itself - or even to call out to an interactive shell on /dev/tty if you want. But most generally I prefer to link a heredoc to a function definition.

fn() { : do something w/ fd3
} 3<<INPUT
$1
$(:gen some output;:maybe cat or head <stdin)
INPUT
while IFS= read -r line; do fn "$line"; done

It is also possible to read one heredoc another writes.

cat 4<<SAVED <<AND
$(  cat)
SAVED
$(  printf %s\\n some random lines
     cat <&4)
AND

Or you could loop like...

until test && cat <&4
do exec 3<&4 4<<IN
$(: gen output; cat <&3)
IN
done 4<infile

And of course you could call something like the fn above from within as well.

And regarding the trailing blank line problem - if it is a problem - then you can simply echo . at the tail end of the command sub, then make sure you strip the last line from the fd when you read it like sed \$d <&"$fd".

But, while pretty secure, looping over files like this in the shell is usually a bad idea anyway - notice there is at least a fork per generated file? Shells assign fds and utilities handle them - do your loop in a sed or awk script and manipulate the file data w/ a standard utility then use the shell to direct it on output - that is usually best practice.