is 'sed' thread safe

Question

If I have a shell/python script that uses sed to modify a file in place based on user inputs, and then two users run the same script at the same time or approx. same time, is 'sed' thread safe ? Or perhaps it is not an issue because the file_descripor that was opened by the first thread will be used to lock the file anyway ? thx

Answer 1

I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:

$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY)               = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foo\n", 4096)                  = 4
...
write(4, "fee\n", 4)                    = 4
read(3, "", 4096)                       = 0
...
close(3)                                = 0
close(4)                                = 0
rename("./sedDe80VL", "foo.txt")        = 0

At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.

Notes:

This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).

Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.

perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.

Answer 2

It is not recommendable to use sed -i on the same file on multiple parallel task, but if you have to...

The pattern to make it work is by using a Semaphore.

To allow asynchronous call on sed -i on the same file in a functional thread aware way is to use Advisory lock.

flock theSharedFile sed -i s/"userInput1"/"$userInput2"/g theSharedFile

Flock creates and advisory lock on theSharedFile, and will execute the sed command with the rest of the arguments when the lock is free (when no other flock is running on theSharedFile).

If two different users trigger it at same time, the second one will wait until the previous ones end. Like that it will be a fully functional and thread safe approach. No "user" change will be discarded overridden.

Since it is an advisory lock, it will only be significant for flock, and should not cause any kind of deadlock. Anyone reading the file will never get locked. Notice if file is large it can be expected to take significant amount of time of waiting for the processes waiting for the lock.

You can list current locks with lslocks

Hope this helps someone out there. Thanks for your time.