0

I have a script that copies a local file to a remote directory and updates a remote file with a new line that mentions the just copied script.
Some times some of the files in the remote directory need to be deleted and the corresponding file mentioning them should be updated to delete corresponding entries as well.
Problem: The script can run from more than one instances and for more than one directory in the remote server. I.e. there are more than 1 directories in the remote server that store files that we copy based on the type passed to the instance. How can I make the copy of the file and the update of the “log” somewhat atomic?
I don’t have so much concurrent instances running for it to be a serious problem but I was wondering if there is a way to do such changes making sure the file is updated correctly.
E.g would the following work? 

scp file.bin remoteserver:/foo/$type/  
grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
echo "$record" >> entries.log
Jim
  • 10,120

1 Answers1

2

You need to use a lock; if you’re running Linux and have flock (part of util-linux), it’s as simple as

scp file.bin remoteserver:/foo/$type/
(
  flock 9
  grep -v "oldfile.bin" entries.log > entries.log.backup && mv entries.log.backup entries.log  
  echo "$record" >> entries.log
) 9>entries.log.lock

This will copy the file, wait to acquire the lock, then update the log exclusively. I’m assuming you wanted to append to the log.

The idea here is to use entries.log.lock as a lock file (we can’t use the log file itself since we’ll be replacing it). 9>entries.log.lock creates the file if necessary and associates file descriptor 9 to it, in the subshell which will run the commands between parentheses. flock 9 acquires a lock on the file pointed at by file descriptor 9, waiting if necessary; the lock will remain held until the file descriptor is closed, which will happen when the subshell exits. grep, mv and echo then proceed under the lock.

Concurrent processes will be able to scp in parallel, but the log manipulation will be serialised.

If you don’t have flock, What Unix commands can be used as a semaphore/lock? has various alternatives (the mkdir approach will work everywhere).

Stephen Kitt
  • 434,908
  • Yes append. But could you explain a bit the syntax and what's going on? What is 9>entries.log.lock? – Jim Oct 18 '17 at 08:05
  • Also is it possible the copy and the update to be both atomic? I.e. other instances wait for that copy & update to finish – Jim Oct 18 '17 at 08:12
  • If you want the copy to be atomic, you can move the scp inside the subshell, after flock. In that case you could perhaps use the last pattern given in the manpage instead, to lock the whole shell script. – Stephen Kitt Oct 18 '17 at 08:13
  • Where does the 9 for file descriptor come from? Could I end up not being able to use that number? Is it arbitrary? – Jim Oct 18 '17 at 08:20
  • Also the last pattern you are referring is it the section: This is useful boilerplate code for shell scripts. Put it at the top of the shell script you want to lock... – Jim Oct 18 '17 at 08:21
  • (Quick clarification: copy atomicity isn’t addressed here, I should have said “protected by the lock” in the comment above.) 9 comes from the example in the manpage. It’s arbitrary, pretty much guaranteed to work, and many other values would work — you’re only competing with the other file descriptors open in the shell (typically, 0, 1, 2 and some other high value for the terminal in some shells). The last pattern I’m referring to is indeed that section. – Stephen Kitt Oct 18 '17 at 08:24
  • Is flock available on all platforms? I get No manual entry for flock when I do man flock. – Jim Oct 18 '17 at 08:31
  • What would be an alternative then? – Jim Oct 18 '17 at 08:38
  • See my update and https://unix.stackexchange.com/q/70/86440. – Stephen Kitt Oct 18 '17 at 08:45
  • If I understood correctly , this won't prevent other programms/processes from updating the entries.log at same time. (though, it may not be required) – RomanPerekhrest Oct 18 '17 at 11:21
  • @Roman yes, such locking is cooperative (see this answer to a related question). – Stephen Kitt Oct 18 '17 at 11:26