Redirect stderr to a file, but create only if any stderr happened?

Question

I'm running a command like:

parallel --spreadstdin --line-buffered 'some_command 2> `mktemp --tmpdir /tmp/stderr`' | do_something

The trick is that parallel creates a lot of processes and they all get a stderr file, most of which is uninteresting because they're empty. How can I make my shell generate a stderr file only if any standard error output actually happened?

I'd prefer to monitor logs on the fly and empty files make it less convenient. — d33tah, Oct 03 '14 at 12:06
@don_crissti: I'm not exactly satisfied by this solution, but thanks anyway. — d33tah, Oct 03 '14 at 13:39

score 1 · Answer 1 · edited Aug 05 '15 at 17:39

As far as I know, this is impossible. The shell reads right to left, so if 2> error exists. an error file will be created. I don't think there's any way around that.

You might be able to do some complex tricks such as using a named pipe and testing its output but I really don't think that's worth it. Instead, I would either use inotify as explained in another answer or create tmp files and copy them to a different location if they contain data.

tmp=$(mktemp); command 2> "$tmp"; [ -s "$tmp" ] && cp "$tmp" /path/to/log/files

score 0 · Answer 2 · edited Jun 11 '20 at 12:04

0

parallel --spreadstdin --line-buffered 'some_command 2 \

mktemp --tmpdir /tmp/stderr' | do_something && if [test -s /tmp/stderr]; then; sleep .1; else; rm /tmp/stderr

One-Liner to Test if File Exists and Size is Greater than Zero for TRUE; empty if FALSE.

I.T. is about doing something that works. Stop counting your bits and bytes, and create an inotifywait trigger on startup to check your logs if empty, record results to datatable, and wipe log on complete; then program a web interface that emails you when keywords appear in your error log.

while:
  do
    inotifywait -e create /tmp/* && \
      if [ -s /tmp/stderr ]
        then
        ##put it in a datatable
        ##remove the log
      fi
  done

iNotifyWait can also monitor Kernel F/S subsystem to see if a file is changed, if you are appending. Otherwise, modify the Source Code and recompile to prevent the error from being created by modifying the try() catch() or error reporting section (the part that prints error codes to the screen).

edited Jun 11 '20 at 12:04

Community

1

answered Oct 03 '14 at 12:29

Tyler Maginnis

1,445

I'd expect it to be more complex in the working case since you'd have to be looking at a particular temporary file and as I said, I'd rather prevent empty stderr files from being created at all. – d33tah Oct 03 '14 at 12:32
+1 inotifywait; -1 "Stop counting your bits and bytes ... program a web interface that emails you when keywords appear in your error log"; -2 "prevent errors" - there's nothing wrong about error reporting on stderr, it usually isn't a bug in the software that is reported there. – peterph Oct 03 '14 at 13:24
@peterph: in my case it is usually some kind of a bug. – d33tah Oct 03 '14 at 13:40
@d33tah if you think you have a bug, wouldn't it be easier to fix the bug, than to build a logging facility? Use trap to process errors. – eyoung100 Oct 03 '14 at 14:10
First you need to trace down the bug. – d33tah Oct 03 '14 at 14:50
@d33tah I wouldn't really consider a bug in the program something that doesn't crash it. When it manages to report it on stderr, it often is a corner case treated by a fallback path. Unless the error is Segmentation fault or similar of course. – peterph Oct 03 '14 at 15:55

score 0 · Answer 3 · edited May 23 '17 at 12:40

It’s amazing how close the answer to that other question on SO came to answering the question without hitting the mark. Define a shell function (or write a script) like this:

save_err()
{
    # Check that "$1" is set and either complain/abort or generate a default.
    if read -r x
    then
        { printf "%s\n" "$x"; cat; } > "$1"
    fi
}

Then use as:

some_command 2> >( save_err $(mktemp --tmpdir /tmp/stderr) )

Caveats: This will fail if the stderr output is a single line not ending with a newline. To fix that, change the guts of the function to

    if read -r x
    then
        { printf "%s\n" "$x"; cat; } > "$1"
    elif [ "$x" != "" ]
    then
        printf "%s" "$x" > "$1"
    fi

I used printf rather than echo to guard against text beginning with a dash or containing backslashes. There may be issues if the first line of the stderr output contains non-ASCII characters.

Redirect stderr to a file, but create only if any stderr happened?

3 Answers3