stderr gets flushed before stdout, when using file logger

Question

My python code:

import sys
print "i am a daemon"
print "i will be run using nohup"
sys.stderr.write("i am an error message inside nohup process\n")

When i run the code as python a.py, it shows,

i am a daemon
i will be run using nohup
i am an error message inside nohup process

When i run the code as nohup python a.py > a.log 2>&1 < /dev/null & , a.log shows,

i am an error message inside nohup process
i am a daemon
i will be run using nohup

Why does stderr logs get flushed/written before stdout logs when using nohup?

Petr Skocik · Accepted Answer · 2015-05-29T09:53:55.273

I don't think it's got anything to do with nohup. You get the same behavior when you do python a.py > a.log 2>&1.

Python is most likely using C file stdio underneath. With that, stdout, when in a terminal, will be line-buffered, and buffered when stdout is a file. stderr is always unbuffered.

Redirecting stdout to a file will switch stdout's buffering from line-buffered to buffered and cause the printed string to be stuck in the buffer, which only gets flushed when your program (the stream) closes. The stderr stream makes it to the file faster because it's unbuffered.

You can use stdbuf to tweak standard buffering, forcing the lines to print in the correct order:

stdbuf -o0 python a.py  >a.log 2>&1

score 2 · Answer 2 · edited Apr 13 '17 at 12:37

This is normal behavior for output streams in most languages: they are buffered, i.e. write actually writes to a buffer in memory and this buffer is written out to the stream in batches. Stdout is line buffered when writing to a terminal (i.e. an actual write takes place every time a newline is printed) but fully buffered (the data is written to memory until the memory buffer gets full) when writing to a regular file or a pipe. Stderr is either unbuffered or line buffered.

In Python, you can choose the type of buffering when opening a file, but not for the standard streams. If you want all streams to be unbuffered, you can set the PYTHONUNBUFFERED environment variable to force the standard streams to be unbuffered. Alternatively, you can run the program under stdbuf or unbuffer.

But if your program doesn't emit output in an appropriate order when stdout is redirected, that's a defect in the program, which you should fix. In particular, if you're going to emit an error message that concerns output written to stdout, you should flush stdout first:

print some_data
if errro_condition():
    file.stdout.flush()
    sys.stderr.write('Bad stuff happened\n')

stderr gets flushed before stdout, when using file logger

2 Answers2

Linked