4

I have a program long_interactive_script.py which has thousands of print statements. I want to pipe the program through tee (or an alternative) so that I can save the output.

If I do

long_interactive_script.py | tee logfile.txt

Python puts its print statements in a 4K buffer, causing me to get:

nothing, nothing, nothing, nothing, a whole lot of text!, nothing, nothing, a sudo prompt in the middle of a word, nothing, nothing, a whole lot of text!

In an attempt to avoid the buffer I tried:

unbuffer long_interactive_script.py | tee logfile.txt

But this causes my script to stop being interactive. So when the script breaks into a sudo prompt, it halts.

Note: I cannot simple sudo BEFORE running the script. The interactive script only requires sudo on some runs, and I don't want to ask for sudo when it isn't necessary.

More...

stdbuf -oL long_interactive_script.py | tee -a logfile.txt

works to some extent. I get all the desired data, but I also get this error:

ERROR: ld.so: object '/usr/lib64/coreutils/libstdbuf.so' from LD_PRELOAD cannot be preloaded: ignored.
Sunny
  • 41
  • 3
  • "stdbuf -oL long_interactive_script.py | tee -a logfile.txt" works to some extent. I get all the desired data but I also get this error "ERROR: ld.so: object '/usr/lib64/coreutils/libstdbuf.so' from LD_PRELOAD cannot be preloaded: ignored." – Sunny Sep 23 '16 at 14:13
  • Please update the question instead of posting auxiliary comments. –  Sep 23 '16 at 14:20
  • 1
    Python should have a means to unbuffer a filehandle, such as $fh->autoflush in Perl, or fconfigure stdout -buffering line in TCL. This would avoid the complications imposed by stdbuf. – thrig Sep 23 '16 at 15:19
  • Interesting issue with stdbuf. Does a simple stdbuf -oL true produce the error? Also you might try python -u long_interactive_script.py | tee -a logfile.txt – Pádraig Brady Sep 23 '16 at 16:19
  • @Brady, a simple stdbuf -oL true does not produce the error. Nor does something like stdbuf -oL echo "hi" | tee /tmp/logfile.txt – Sunny Sep 26 '16 at 14:20

2 Answers2

1

Specify a zero-sized buffer for Python's standard output stream. You can do this by invoking Python with the -u flag, or with the following statement.

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Diomidis Spinellis
  • 722
  • This works great in terms of getting all the output, but now my command line hangs even though long_interactive_script.py has finished executing. long_interactive_script.py also launches executable_A and executable_B. For some reason the command doesn't finish until I close the executables. – Sunny Sep 26 '16 at 14:17
  • Unless the commands execute asynchronously (e.g. with & on a Unix shell), this is the expected behaviour. – Diomidis Spinellis Sep 26 '16 at 19:09
1

Workarounds for buffering notwithstanding, that is why others use script, which captures all of the characters written to the terminal without interfering with interactive prompts.

The resulting typescript file is a little uglier than just redirecting to the standard output, but (assuming that your application was providing useful information in that mode), just trimming carriage return characters will suffice.

Beyond just carriage returns, there are editing characters to filter out. See for example:

Community
  • 1
Thomas Dickey
  • 76,765