What Unix commands can be used as a semaphore/lock?

Question

I want to run multiple Bash shell scripts in parallel. However, I want to avoid race conditions. What Unix commands are truly atomic that I could use for this purpose, and how can I use them?

What are you doing that requires parallel work? Can't you express the dependencies in a way that allows a parallel make(1) to take over? (i.e., do a make -j 9 if you have 8 cores)? This has the added advantage of interleaving work with finer granularity. — vonbrand, Mar 15 '13 at 14:50
See also Quick-and-dirty way to ensure only one instance of a shell script is running at a time on stackoverflow. — Tim Bunce, Mar 18 '15 at 13:26
Related: https://stackoverflow.com/questions/6870221/is-there-any-mutex-semaphore-mechanism-in-shell-scripts — Roel Van de Paar, Jun 11 '21 at 09:28
The Linux Kernel documentation lists system calls (of file-system commands) applying semaphore as locking mechanism (see inode_operations section). — Ham, Sep 13 '22 at 09:00

score 39 · Answer 1 · answered Aug 11 '10 at 13:41

39

flock(1)

#!/bin/bash

# Makes sure we exit if flock fails.
set -e

(
  # Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
  flock -x -w 10 200

  # Do stuff

) 200>/var/lock/.myscript.exclusivelock

This ensures that code between "(" and ")" is run only by one process at a time and that the process does wait for a lock too long.

answered Aug 11 '10 at 13:41

Alex B

4,478

Nice one, didn't know about it. However, it's apparently Linux-specific... – Riccardo Murri Sep 07 '10 at 19:52
1

@Riccardo, FreeBSD has a similar command: lockf(1). – Alex B Sep 08 '10 at 11:08
lockf(1) doesn't work in the way used in this example, though. It can't take a file descriptor number as an argument. – Charley Oct 28 '16 at 07:36
@RiccardoMurri brew install discoteq/discoteq/flock – HappyFace May 26 '20 at 13:14
1

Shorter but equivalent: flock -w 10 /var/lock/.myscript.exclusivelock -c '# do stuff'. No need for ( ... ) 200> ... if you only execute a few simple commands. However, if you want to modify the environment (for instance, assign a variable inside the lock for later use) then use { ... } 200> .... – Socowi Sep 10 '20 at 21:38
this is the correct answer - flock is available on all linux distros and as @HappyFace commented, available as ok'ish port for bsd – christian elsee Jan 19 '21 at 10:26
Reading the man page I can't find the reason why it's specifically "200" ? What does 200 means ? I know that 10 is the number of the seconds to wait before failing – Freedo May 10 '23 at 07:03
Also I just tested and if the script is killed with -9 then the script can never run again, unless you manually remove the lock ? – Freedo May 10 '23 at 07:05

score 37 · Accepted Answer · edited Jan 27 '15 at 17:12

37

If lockfile is not installed on your system, then mkdir will do the work: it's an atomic operation, and it fails if the directory already exists (as long as you don't add the -p command-line switch).

create_lock_or_wait () {
  path="$1"
  wait_time="${2:-10}"
  while true; do
        if mkdir "${path}.lock.d"; then
           break;
        fi
        sleep $wait_time
  done
}

remove_lock () {
  path="$1"
  rmdir "${path}.lock.d"
}

edited Jan 27 '15 at 17:12

waitinforatrain

1,685

answered Aug 11 '10 at 12:14

Riccardo Murri

16,468

1

10 years late, but the correct answer then, and now ,is flock, discussed below (https://unix.stackexchange.com/a/291/343121) – christian elsee Jan 19 '21 at 10:30

score 12 · Answer 3 · answered Aug 10 '10 at 21:04

lockfile(1) looks like a good candidate, though beware that it's part of the procmail package, which you may not have installed on your machine yet. It's a popular enough package that it should be packaged for your system if it's not installed yet. Three of the four systems I checked have it, and the other has it available.

Using it is simple:

#!/bin/sh
LOCKFILE=$HOME/.myscript/lock
mkdir -p `dirname $LOCKFILE`

echo Waiting for lock $LOCKFILE...
if lockfile -1 -r15 $LOCKFILE
then
    # Do protected stuff here
    echo Doing protected stuff...

    # Then, afterward, clean up so another instance of this script can run
    rm -f $LOCKFILE
else
    echo "Failed to acquire lock!  lockfile(1) returned $?"
    exit 1
fi

The options I've given make it retry once a second for up to 15 seconds. Drop the "-r" flag if you want it to wait forever.

Just for reference - the man page: http://linux.die.net/man/1/lockfile. :) — Lucas Jones, Aug 10 '10 at 21:50
Be aware that (according to the manpage), "Once a file is locked, the lock must be touched at least once every five minutes or the lock will be considered stale, and subsequent lock attempts will succeed." — Jay, Jan 05 '19 at 19:19

score 9 · Answer 4 · edited May 04 '13 at 20:10

9

The system call mkdir() is atomic on POSIX filesystems. So, using the mkdir command in such a way that it involves exactly one call to mkdir() would achieve your purpose. (IOW, don't use mkdir -p). The corresponding unlock is rmdir of course.

Caveat emptor: mkdir() might not be atomic on network filesystems.

edited May 04 '13 at 20:10

Anthon

79,293

answered Aug 11 '10 at 15:15

Hari

311

2

is rmdir therefore also atomic? – Alexej Magura May 11 '17 at 23:20

score 2 · Answer 5 · edited Sep 02 '14 at 19:02

2

If you're only running on Unix, use fifos. You can write work records to the fifo and have processes read from this file and your readers will block on the fifos.

Lock files are ok, but for what you describe I would go with fifos

edited Sep 02 '14 at 19:02

Michael Mrozek

93,103
40
240
233

answered Sep 02 '14 at 18:54

Chris

21

2

Could you elaborate on how you think fifos can prevent race conditions? – G-Man Says 'Reinstate Monica' Sep 02 '14 at 19:24

jacksonh · Answer 6 · 2010-08-10T21:06:36.667

2

Maybe the lockfile command will do what you need.

lockfile ~/.config/mylockfile.lock
.....
rm -f important.lock

edited Aug 10 '10 at 21:06

answered Aug 10 '10 at 20:05

jacksonh

3,734
1
20
13

3

This seems to delete the wrong file. – Benjamin W. Jul 13 '19 at 19:19

score 1 · Answer 7 · edited Apr 13 '17 at 12:36

As posted here: " Correct locking in shell scripts? ", using FLOM (Free LOck Manager) tool, serializing commands and shell scripts becomes as easy as running

flom -- command_to_serialize

FLOM allows you to implement more sofisticate use cases (distributed locking, readers/writers, numeric resources, etc...) as explained here: http://sourceforge.net/p/flom/wiki/FLOM%20by%20examples/

What Unix commands can be used as a semaphore/lock?

7 Answers7

Linked

Related