Dynamic file content generation: Satisfying a 'file open' by a 'process execution'

Question

I have a read-only file, F.

A program, P, that I'm not the author of, needs to read F.

I want the content of F to come from another 'generator' program, G, whenever P tries to read F (taking F to be an ordinary file) and not any earlier.

I tried doing the following:

$ mkfifo /well-known/path/to/F    # line #1
$ G > /well-known/path/to/F       # line #2

Now, when P starts up and tries to read F, it appears to be able to read the output generated by G just as I wished it to. However, it can do so only once, since G after all gets to run only once! So, if P had a need to read F again later in its execution, it would end up blocking on the fifo!

My question is, other than bracketing line #2 above in some sort of an infinite loop, is there any other (elegant) alternative for the above?

What I'm hoping for is, some way of registering a 'hook' program into the file-open system call such that the file-open would result in the launching of the hook-program and the file-read in the reading of the hook-program output. Obviously the assumption here is: the read will happen sequentially from file beginning to file end, and never in random seeks.

Answer 1

FUSE + a soft-link (or a bind mount) is a solution, though I would not consider it "elegant", there's quite a lot of baggage. On *BSD you'd have the simpler option of portalfs, with which you could solve the problem with a symlink – there was a port of it to Linux many years ago, but it seems to have been dropped, presumably in favour of FUSE.

You can quite easily inject a library to override the required open()/open64() libc call(s) that it makes. e.g.:

#define _GNU_SOURCE
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <dlfcn.h>
#include <stdarg.h>

//  gcc -Wall -rdynamic -fPIC -nostartfiles -shared -ldl -Wl,-soname,open64 \
//       -o open64.so open64.c

#define DEBUG 1
#define dfprintf(fmt, ...) \
    do { if (DEBUG) fprintf(stderr, "[%14s#%04d:%8s()] " fmt, \
          __FILE__, __LINE__, __func__, __VA_ARGS__); } while (0)

typedef int open64_f(const char *pathname, int flags, ...);
typedef int close_f(int fd);
static  open64_f   *real_open64;
static  close_f    *real_close;

static FILE *mypipe=NULL;
static int mypipefd=-1;

//void __attribute__((constructor)) my_init()
void _init()
{
    char **pprog=dlsym(RTLD_NEXT, "program_invocation_name");
    dfprintf("It's alive! argv[0]=%s\n",*pprog);

    real_open64 = dlsym(RTLD_NEXT, "open64");
    dfprintf("Hook %p open64()\n",(void *)real_open64);
    if (!real_open64) printf("error: %s\n",dlerror());

    real_close = dlsym(RTLD_NEXT, "close");
    dfprintf("Hook %p close()\n",(void *)real_close);
    if (!real_close) printf("error: %s\n",dlerror());
}

int open64(const char *pathname, int flags, ...)
{
    mode_t tmpmode=0;
    va_list ap;
    va_start(ap, flags);
    if (flags & O_CREAT) tmpmode=va_arg(ap,mode_t);
    va_end(ap);

    dfprintf("open64(%s,%i,%o)\n",pathname,flags,tmpmode);

    if (!strcmp(pathname,"/etc/passwd")) {
        mypipe=popen("/usr/bin/uptime","r");
        mypipefd=fileno(mypipe);
        dfprintf("  popen()=%p fd=%i\n",mypipe,mypipefd);
        return mypipefd;
    } else {
        return real_open64(pathname,flags,tmpmode);
    }
}

int close(int fd)
{
    int rc;
    dfprintf("close(%i)\n",fd);
    if (fd==mypipefd) {
        rc=pclose(mypipe); // pclose() returns wait4() status
        mypipe=NULL; mypipefd=-1;
        return (rc==-1) ? -1 : 0;
    } else  {
        return real_close(fd);
    }
}

Compile and run:

$ gcc -Wall -rdynamic -fPIC -nostartfiles -shared -ldl -Wl,-soname,open64   \
    -o open64.so open64.c 
$ LD_PRELOAD=`pwd`/open64.so cat /etc/passwd
19:55:36 up 1110 days,  9:19, 55 users,  load average: 0.53, 0.33, 0.29

Depending on exactly how the application works (libc calls), you may need to handle open() or fopen()/fclose() instead. The above works for cat or head, but not sort since it calls fopen() instead (it's straightforward to add fopen()/fclose() to the above too).

You probably need more error handling and sanity checking than the above (especially with a long running program, to avoid leaks). This code does not correctly handle concurrent opens.

Since a pipe and a file have obvious differences, there is a risk that the program will malfunction.

Otherwise, assuming you have daemon and socat you can pretend you don't have an infinite loop:

daemon -r -- socat -u EXEC:/usr/bin/uptime PIPE:/tmp/uptime    

This has the slight disadvantage (which should be evident here) of the provider program starting to write then blocking, so you see an old uptime, instead of it being run on-demand. Your provider would need to use non-blocking I/O in order to properly provide just-in-time data. (A unix domain socket would allow a more conventional client/server approach, but that's not the same as a FIFO/named pipe that you can just drop in.)

---

Update see also this later question which covers the same topic, though generalised to arbitrary processes/readers rather than a specific one: How can I make a special file that executes code when read from (note that fifo-only answers won't reliably generalise to concurrent reads)

Answer 2

What you could do is having a daemon (let's call it filld as it fills up F each time P needs it).

When you launch it, it tries to open the FIFO (and blocks as there is no reader). Each time a reader comes, it writes to the FIFO what it needs to write (fork-ing and exec-ing G for example), closes the FIFO and reopens it. Each time, it succeeds in opening, it writes to F. If it receives a SIGPIPE, it closes everything, and blocks itself on the FIFO again.

Answer 3

An infinite loop is the reason for at least two reasons. Beside the one you noticed: You want the file updated on every access, don't you? Otherwise you could simply create a regular file with the right content.

A hook for open*() is possible but probably not trivial. FUSE is the way to go. If you are lucky then a module for this already exists. Otherwise you need to write your own module which hooks this one path and passes the rest through.