Linux: Writing a watchdog to monitor multiple processes

Question

A few years ago, a coworker came up with an elegant solution for a watchdog program. The program ran on Windows and used Windows Event objects to monitor the process handles (PID’s) of several applications. If any one of the processes terminated unexpectedly, its process handle would no longer exist and his watchdog would immediately be signaled. The watchdog would then take an appropriate action to “heal” the system.

My question is, how would you implement such a watchdog on Linux? Is there a way for a single program to monitor the PID’s of many others?

See also Ensure a process is always running for a list of monitoring/supervising programs. — Gilles 'SO- stop being evil', Jun 06 '11 at 21:47
Also see stackoverflow: http://stackoverflow.com/questions/9400724/foolproof-cross-platform-process-kill-daemon — Mike Pennington, May 01 '12 at 18:49

score 17 · Accepted Answer · answered Feb 18 '11 at 21:34

The traditional, portable, commonly-used way is that the parent process watches over its children.

The basic primitives are the wait and waitpid system calls. When a child process dies, the parent process receives a SIGCHLD signal, telling it it should call wait to know which child died and its exit status. The parent process can instead choose to ignore SIGCHLD and call waitpid(-1, &status, WNOHANG) at its convenience.

To monitor many processes, you would either spawn them all from the same parent, or invoke them all through a simple monitoring process that just calls the desired program, waits for it to terminate and reports on the termination (in shell syntax: myprogram; echo myprogram $? >>/var/run/monitor-collector-pipe). If you're coming from the Windows world, note that having small programs doing one specialized task is a common design in the Unix world, the OS is designed to make processes cheap.

There are many process monitoring (also called supervisor) programs that can report when a process dies and optionally restart it and far more besides: Monit, Supervise, Upstart, …

+1, exactly what I was looking for - thank you for such a detailed answer. — Justin Ethier, Feb 20 '11 at 02:56

score 0 · Answer 2 · answered Feb 21 '11 at 04:43

My approach to this issue is to use init and its built-in respawn directive to start/restart whatever you need to run. This was its original intent and main purpose. In some cases you will need to run a script to cleanup after a process has died, or to prepare for the process to start (most of the time the work is the same). In most cases a bash script that ends in exec works great for this.

Linux: Writing a watchdog to monitor multiple processes

2 Answers2