7

If I have 100 servers execute a command, and each redirects their output to the same file on an NFS share, will the resultant file be interleaved neatly or can it become corrupt?

For example, if this command was executed on 100 servers:

find / type -f >> /some_server/share/file_list.txt

Would it be corrupt?

ps. The command is just an example, I'm not actually listing files

Gilles 'SO- stop being evil'
  • 829,060
Fidel
  • 405
  • With that command each one will overwrite the other and the last machine to run the command will be the one with the results. – Stephen Harris Aug 01 '16 at 14:18
  • Seems like a reasonable Answer, Stephen - want to write something up? – Jeff Schaller Aug 01 '16 at 14:27
  • The title talks about appends, which would also make a far more interesting question... – ilkkachu Aug 01 '16 at 15:06
  • The title says "appenders", the body asks if it will be "interleaved neatly" or "corrupted", and the command is an overwrite > versus append >>. Fidel, what is your goal? Appended output? Interleaved output? Is your actual command using > or >>, or some other "appending" program? – Jeff Schaller Aug 01 '16 at 15:10
  • Apologies guys, it is indeed >> append – Fidel Aug 01 '16 at 17:05

1 Answers1

10

In short: Yes, simultaneous writes from multiple NFS clients will be corrupted.

Simultaneous appends locally are nicely interleaved, since the OS knows the file is opened in append mode, and atomically seeks to the current end of file before each write call, regardless of other processes that may have extended the file in the meanwhile.

But on NFS, no such luck. The Linux NFS FAQ states it plainly:

A9. Why does opening files with O_APPEND on multiple clients cause the files to become corrupted?
A. The NFS protocol does not support atomic append writes, so append writes are never atomic on NFS for any platform.
Most NFS clients, including the Linux NFS client in kernels newer than 2.4.20, support "close to open" cache consistency,

A8. What is close-to-open cache consistency?
A. Perfect cache coherency among disparate NFS clients is very expensive to achieve, so NFS settles for something weaker that satisfies the requirements of most everyday types of file sharing. [...]

When the application closes the file, the NFS client writes back any pending changes to the file so that the next opener can view the changes. This also gives the NFS client an opportunity to report any server write errors to the application via the return code from close(). This behavior is referred to as close-to-open cache consistency.

The NFS write operations just contain a position to write to, and the data to be written. There's no provision for centrally coordinating where the end of file is, which is what would be needed to make sure clients don't overwrite each other.

(The page does seem a bit old, since it talks mostly about Linux kernel version 2.6, with no mention of 3.x. NFS version 4 is however mentioned in relation to kernel support, so it could be assumed the answer applies to v4 also.)

A little test made on a recent-ish Linux and an NFS v3 share:

Writing numbers out in a shell loop (for ((i=0 ; i<9999 ; i++)) ; do printf "$id %06d\n" $i >> testfile ; done), on two clients at the same time results in nicely corrupted output. Part of it:

barbar 001031
foo 000010
32
foo 000011
33
foo 000012

Here, the first loop on one machine wrote lines with barbar, while another loop on another machine wrote the foo lines. The line that should say barbar 001032 is written starting at the same position as foo 000010 line, and only the final numbers of the longer line are visible. (Note that in this case the file is actually opened and closed for each printf since the redirection is inside the loop. But it only helps in finding what the end of file was when the file was opened.)

If the file is kept open the whole time, larger blocks may be overwritten, since the promise is only that the client system writes changes to the server when the file is closed. Even truncating the file when opening doesn't change this much, since the truncation only clears the file, but does not prevent further writes by the other client when it closes the file.

Community
  • 1
ilkkachu
  • 138,973
  • Does this apply if the clients are on the same machine? i.e. if you would normally be writing from multiple processes on one machine in chunks of less than PIPE_BUF, does using an NFS mount on that machine break the guarantee? – Sam Brightman Dec 21 '16 at 11:26
  • I know this is old but a quick question, if programs sync their writes so that there is no overlapping writes happening would it still corrupt? I mean program 1 is writing first 500MB the next program on another machine is writing next 500MB when the server syncs will it corrupt? i am hoping not – Laukik Dec 31 '22 at 15:57
  • @Laukik, all writes should go where the write order says, so if the clients know what they're doing, it shouldn't cause any corruption. It's just that when appending, the position where the write should go can change behind a client's back. – ilkkachu Dec 31 '22 at 17:03
  • But those sorts of writes don't tend to happen except in some kind of binary database-like files, and those might face other isues, as there's no guarantee a write is immediately visible to other clients – ilkkachu Dec 31 '22 at 17:06
  • Ohh thats a bummer, so nfs does not guarantee write visible to a read, even if we close or open file post read or write?, the reason for asking is cause we are desiging a database based on files, but block storage is costly so moving to EFS is cheaper but it has its own share of issues.. – Laukik Jan 01 '23 at 05:25
  • @Laukik, well, based on what I've quoted from the docs above, closing and opening should help. If there's something elde there too, I don't know. Anyway I was thinking that a lot of applications might not close and reipen in the middle if their work. On a local system, a write should be immediately visible to other readers, without reopening. – ilkkachu Jan 01 '23 at 09:10