How to capture stderr of function that modifies current environment?

Question

The module function of the Environment Modules¹ package does its work by modifying various environment variables of the current shell process.

Unfortunately, this function returns 0 whether it succeeds or not², which makes it difficult for a client script to respond appropriately to failures.

I would like to implement a function wrapper mymodule around module that passes all its arguments straight through to module, and properly returns a non-zero value if module fails.

The only way that mymodule can detect whether module failed is to inspect the output module function writes to stderr, if any.

The problem is that I can't come up with a reasonable way for mymodule to get this output without nullifying module's actions. More specifically, almost all the ways I can think of to capture module's stderr into a variable entail running module in a child process, thus preventing it from doing its job (which requires modifying the current shell).

The one exception to the above would be to redirect module's stderr to a temporary file, but I hate the idea of creating a file every time the module function runs.

Is there some way for mymodule to invoke module in the current environment and at the same time capture its stderr in a variable?

I am interested in answers for both zsh and bash.

---

^{1 Not to be confused with the Lmod environment modules package, which has a very similar interface.}

^{2 At least this is the case for the ancient version 3.2.9 that I must work with. I have no control over this.}

Answer 1

Both Bash and zsh have coprocesses (unfortunately slightly different), which essentially wrap up a pipe call and spawning a subprocess with both its standard input and standard output available to the calling shell. In essence they let the shell play both x and y in x | cmd | y, but with all of x, y, and cmd running from the current process and not in a pipeline execution environment.

That will let us run module 2>&... for some ..., with module running from the current shell. If we use cat as our coprocess (i.e. cmd), it will just repeat everything straight back out again, and then we can read the output back into the current shell again with y <&... later on.

Another option is to redirect standard error into another background process and wait for its return code. I will address both below.

---

I'm going to use this fake module function to test with so that I can turn errors on and off at will. If modifies the current shell environment so that we can see it and outputs "err" to stderr; I'll be commenting that line in and out as needed:

module() {
        sleep 1
        FOO=$(date)
        echo err >&2
} 

---

If I run a cat coprocess in Bash I can redirect module's stderr into that, and read from cat's stdout to do whatever I want:

coproc cat
module 2>&${COPROC[1]}
exec {COPROC[1]}>&-
if grep -q err <&${COPROC[0]}
then
        echo got an error
else
        echo no error
fi

In zsh it needs to be

module 2>&p
exec 4<&p
coproc :
if grep -q err <&4

in the middle instead.

In either case, I can both run the module command in the current shell and read from the error output there. A function could return within the if as normal.

Everything except cat is running from the current execution environment: FD redirections do not create independent environments the way pipelines do. We can echo $FOO at the end to check that, and see that the date has updated because module ran in the current environment.

---

Alternatively, the background process could just do all the work. This works in Bash:

exec 2> >( if grep -q . ; then exit 7 ; else exit 0 ; fi )
PID=$!
module
exec 2>&-
wait $PID
echo $?

The above will output either 7 or 0 according to what the subprocess at the top said — you can adjust to do whatever you like about the return code. Under zsh it doesn't, because $! isn't set for process substitutions; that ought to be solvable but I stopped trying. A fixed fifo, rather than a temporary file, would also work here.

In this case you'd probably want to save and restore FD 2 on either side as well.

Answer 2

I don't know how "Environment Modules" work, but I'll assume from your description that they're shell functions which set environment variables, and their stderr output should be caught / matched against without running them in a separate process.

The answer, like it or not, is that the only robust and obvious way is to redirect their stderr to a temporary file. Using named pipes is just as unwieldy (you still have to create a temporary file!), in addition to being much trickier. And using co-processes is onerous, awkward and unportable.

In bash (and in bash only) you can take advantage of an undocumented feature ($! being set to the PID from a >(...) process substitution) and get away with something like:

module 2> >(grep error)
wait $! && echo failed

This example assumes that module itself is not spawning any children that may confuse $!.

Answer 3

On Linux, with both bash and zsh, you should be able to do:

my_module() {
  chmod u+w /dev/fd/3 # only needed in bash 5+
  module 2> /dev/fd/3 3>&-
  ! grep -q err /dev/fd/3
} 3<<< ''

The 3<<< '' is a here-string initially containing an empty line. Both zsh and bash implement here-strings and here-documents as deleted temporary files. On Linux (and Cygwin, but generally not other systems), opening /dev/fd/3 opens the file pointed to by the fd 3, even if it's already deleted (on other systems, it duplicates fd 3) so it's a very clean way to work with temporary files there. The file is already deleted, you don't have to worry about its cleanup, and it's only visible on the FS for a very short time (since version 5 however, bash removes write permissions to it which we need to work around with chmod).

Here, you do need a temporary file if you're going to run module and grep in sequence (as opposed to in parallel in separate processes). Doing it with pipes (like in Michael's approaches, but unlike @mosvy's) would lead to dead-locks if there's enough data output that it fills up the pipes.