23

Don't normally post here but I am ripping my hair out over this one. I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes. This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.

Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.

When launched Via systemd, if the parent process exits, the children all exit too (Well, the screens that they launch in die and appear as Dead).

Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?

Thanks!

[Unit]
Description=Server commander
After=network.target

[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid


ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID


[Install]
WantedBy=multi-user.target

Edit:

It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in GNU screens as requested from a central server. It is normally always running, it doesn't spawn services and exit.

There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.

A better explanation of how it works:

  • systemd spawns Server.py
  • Server.py forks and writes the pid file for systemd
  • Server.py then spawns server processes in gnu screen based on its instructions
  • Server.py continues to run to perform any restarts requested from the server

When launching without systemd, Server.py can be restarted and the GNU screens it launches are unaffected. When launching with systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.

AdminBee
  • 22,803
Bottswana
  • 411
  • 1
    It is hard to provide a solution without having the Server.py code and a description of how do the launched services fork (if they fork). However, generally speaking, this is a readiness protocol mismatch problem. – intelfx May 21 '15 at 22:29
  • BTW, the ExecStop= is not needed. systemd's default action on stop is to kill processes. You may want to take a look at the documentation for KillMode= directive. – intelfx May 21 '15 at 22:31
  • 2
    And, finally... If there is no suitable readiness protocol (one of simple or forking, actually), the last resort would be Type=oneshot, RemainAfterExit=yes and KillMode=control-group. – intelfx May 21 '15 at 22:33
  • @intelfx Essentially the Python script launches a server in a screen using Subprocess.call. It's more complicated than that because the script receives commands from elsewhere telling it which screens to start and which to not. Which screens are available is also dynamic hence why they cant be systemd services in their own right. Ideally I dont want systemd to treat these screens as part of the service at all, but currently they get dumped in the same process group and die with the master if its restarted. – Bottswana May 22 '15 at 13:21
  • My hunch is that systemd doesn't "handle" a control process like that (it just looks up PID's at start time, doesn't recognize the later ones...) :| – rogerdpack Jul 09 '19 at 15:30

3 Answers3

18

I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all!

AdminBee
  • 22,803
Bottswana
  • 411
  • That seems like something of a work around more than a fix though, see the other answers...if you do this and do a "systemctl stop" then it won't kill the child processes they'll still be running [?] outside the oversight of systemctl? – rogerdpack Jul 08 '19 at 20:54
  • This is the right solution if you want the child not to be managed by systemd. – Alberto Salvia Novella May 15 '21 at 07:01
7

I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes.

Which indicates that you are doing it wrongly. More in this in a moment.

when the script exits the child processes are orphaned and continue to run.

This is not correct dæmon behaviour. If the "main" process — in this case the child that you have forked, since you have specified Type=forking — exits, systemd considers the service to have deactivated and terminates any other still-running processes (in the control group) in order to tidy up.

Sometimes the conversion from System 5 rc scripts to systemd is not straightforward, because the right way to do things under systemd is quite different. The right way to do (say) OpenVPN, or OpenStack, or OSSEC HIDS in systemd is not the same as one would do it with an rc script. The fact that you have a script that is forking, then spawning a whole load of grandchildren processes, then exiting expecting those grandchildren to keep running indicates that you are perpetrating the same sort of horror as ossec-control, albeit with two less levels of forking. If you find yourself writing a "master" script that checks "enable" flags and runs child processes for the "enabled" parts of your system, then you are making the same mistake as the horrendous ossec-control.

No such home-grown mechanisms are necessary with systemd. It already is a service manager. Per https://unix.stackexchange.com/a/200365/5132, the right way to go about this in systemd is not to have one service that spawns some wacky and confused attempt to have "sub-services". It is to have each child process as a fully fledged systemd service in its own right. Then one enables and disables, and starts and stops, the various parts of the system using the normal systemd controls. As you can see in the OSSEC HIDS case, a simple template service unit covers almost all (one exception is at https://askubuntu.com/a/624871/43344) services, allowing one to do things such as systemctl enable ossec@agentlessd.service to enable an optional agentlessd service, without any need at all for the horrendous "master script" mechanism that was needed with System 5 rc.

There are plenty of cases, not perhaps as extreme as OSSEC HIDS, where such rethinking is necessary. MTSes like exim and sendmail are two such. One might have had a single rc script that spawns a queue runner, an SMTP Submission dæmon, and an SMTP Relay dæmon, with a bunch of ad hoc shell variables in a configuration file to control exactly which are run. But the right way to do this with systemd is to have three proper service units (two of which have associated socket units) and no ad hoc stuff at all, just the regular mechanisms of the service manager.

Community
  • 1
JdeBP
  • 68,745
  • 1
    I appreciate the feedback on this. While I agree having subset services makes sense, it was done in Python for a reason which I can't go into. My only solution is to find a way to make this method work. Thanks, though. I'd love to do it properly. – Bottswana May 22 '15 at 13:15
  • The 'sub'services which the script launches are just servers running in gnu screen as a specific user. These servers change a lot, some are added, some are removed and this is controlled elsewhere, so they can't really be actual services in systemd, because that adds to much complexity, and cant be managed centrally. Also the same script is also used on non systemd servers. – Bottswana May 22 '15 at 13:24
  • systemd has explicit facilities to allow services to be added and removed without needing root access. The "also used on non-systemd services" is the only one of the arguments above that can't be fixed by adding more systemd... well, arguably, even that one could be too. :) – Charles Duffy Jul 02 '19 at 19:21
0

You could just sleep the parent, and wait for systemd to kill it at stop time.

Craig Hicks
  • 664