Why is directing stdout to a file inconsistent and delayed?

Question

I have tried two methods to direct the output of a program's printf calls to a file. In both cases, sometimes the file is updated immediately, sometimes it takes several minutes, and sometimes it doesn't update the file at all.

My program runs from a bash script that is executed by /etc/network/if-up.d/upstart. I have tried this for my bash script:

    #!/bin/bash
    sleep 2

    if ! pgrep "demo" > /dev/null
    then
        sudo /home/pi/code/demo_device/demo >> /home/pi/code/demo_device/auto_log.txt 2>&1 &
    fi

and this:

    #!/bin/bash
    exec 1> >(logger -s -t $(basename $0)) 2>&1
    sleep 2

    if ! pgrep "demo" > /dev/null
    then
        sudo /home/pi/code/demo_device/demo &
    fi

The auto_log.txt file in the first case and the /var/log/user.log file in the second case both have the issues that I described above. I also had the bash script run the demo executable with start-stop-daemon and I got the same results. How can I have a log file update immediately and consistently when running a program as a background process on boot-up?

note: I don't know if this is relevant, but the program is designed to run continuously and never exit.

Answer 1

regarding your output file not updating, try modifying the C program and put a fflush(stdout); statement after your printf() statements. This will force a write to the output stream. Typically if your printf() has a \n in it then it will cause that output to write to the screen or file, but not always. So when you definitely want it to happen then use fflush(); And you don't have to use fflush() after every printf statement, just put it in the place where it makes the most sense, so that at that point you want all previous printf statements which may still be in the buffer to be written out.

And if the C program is not writing to stdout but to some other file handle, then place that file pointer in to fflush() like in the below sample program

#include <stdio.h>

int main ( int argc, char *argv[] )
{
    FILE *fp;

    fp = fopen( "test.txt", "w" );
    if ( fp != NULL )
    {
       fprintf( fp, "hello world\n");
       fflush( fp );

       /* 
          100+ lines of code here doing something else,
          which could cause the above printf to sit in the
          buffer until fclose or the program ends
       */

       fclose( fp );
    }
    return 0;
}

If you can't modify the C program writing the output, then I don't know how to force the buffer to flush from outside the program.