What happens when you read a file while it is overwritten?

Question

Suppose I read (cat) a file while another process is rewriting its contents. Is the output predictable? What would happen?

See this as well: How to make reading and writing the same file in the same pipeline always “fail”?. — codeforester, Apr 04 '19 at 19:21

score 28 · Accepted Answer · answered Jun 27 '12 at 00:40

That depends on what the writer does.

If the writer overwrites the existing file, then the reader will see the new content when the writer overtakes the reader, if ever. If the writer and the reader proceed at variable speeds, the reader may alternatively see old and new content.

If the writer truncates the file before it starts to write, the reader will run against the end of the file at that point.

If the writer creates a new file then moves the new file to the old name, the reader will keep reading from the old file. If an opened file is moved or removed, the processes that have the file opened keep reading from that same file. If the file is removed, it actually remains on the disk (but with no way to open it again) until the last process has closed it.

Unix systems tend not to have mandatory locks. If an application wants to ensure that its writer component and its reader component don't step on each other's toes, it's up to the developer to use proper locking. There are a few exceptions where a file that's open by the kernel may be protected from writing by user applications, for example a loop-mounted filesystem image or an executable that's being executed on some unix variants.

Gilles, does your explanation also apply in ftp/sftp scenarios? Say a process starts reading a transmitted ftp file while another version of the same file is overwriting it due to a new transmission. — iruvar, Aug 04 '14 at 20:27
@1_CR Yes. In that case, the writer and the reader are the ftpd processes. — Gilles 'SO- stop being evil', Aug 04 '14 at 21:46
@VictorChoy Standard meaning: to start behind/after someone else and at some point move ahead of them. — Gilles 'SO- stop being evil', Oct 10 '16 at 17:50

score 23 · Answer 2 · answered Jun 26 '12 at 14:21

It's a classic race condition, so the outcome is unpredictable by definition.

Among others, it depends on

fopen(3) or open(2) write modes,
how/if the writer is buffering its output,
how the reader is reading the file,
the speed difference between the reader and writer,
the time difference between the read and writer's start.
And of course, on modern multi-core machines, things are complicated even more by other factors lower down (e.g. process scheduling).

If you need to be able to read a file while it's being rewritten, then you can make the writer make a transient copy of the file, modify that, then copy it back to the original file. This the way rsync does this, for instance. There are a number of ways to implement this, but no free lunch. Each method has its own shortcomings and repercussions.

This answer is far more comprehensive than mine. Deleting my answer. — killermist, Jun 26 '12 at 14:22

score 0 · Answer 3 · answered Apr 03 '19 at 13:22

Previous responders have more comprehensive explanations than this, but here's a trick that definitely works as well, pretty much doing exactly what he wants:

$ tail -f <filename>

Will show you the end of the file as it's being written. Handy if you want to pipe STDERR to a file but still see it in another terminal window, for example.

What happens when you read a file while it is overwritten?

3 Answers3