systemd service file for a node.js app is not working when pointed at a shell script

Question

Before I mention my problem, I have already checked in most of the questions related to systemd, but I couldn't find a convincing answer. I wrote a nodejs app that runs a server.

const express = require('express');
const app     = express(),
      port    = process.env.PORT || 5000;

    app.get('/' , ( req , res ) => {
        res.send('Hello World')
    })

    app.listen( port , () => {
        console.log(`The server listens on port ${port}`)
    })

Initially, I created a service that runs the server.js and it worked like a charm.

[Unit]
Description=Hello World
After=network.target

[Service]
ExecStart=/usr/bin/node /home/msimou/Desktop/helloWorld/server.js
User=msimou
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

However, I tried the same task by creating .sh files that either start or stop the service and I tried to embedded in the unit file, but for some unknown reason the server is not working. The updated unit file looks like:

[Unit]
Description=Hello World
After=network.target

[Service]
ExecStart=/bin/bash /home/msimou/Desktop/helloWorld/init/startHelloWorld.sh
ExecStop=/bin/bash /home/msimou/Desktop/helloWorld/init/stopHelloWorld.sh
User=msimou
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

I checked in the logs files at /var/log/syslog and journalctl to spot the error but the only thing that I can see is systemd to continuously start and stop my service evety 5 seconds. When I checked using systemctl status helloWorld.service, its said that the service had a success status, however I couldn't find any process related to my nodejs application.

Answer 1

Since you did not specify a service type, systemd will assume the default Type=simple.

As a result, systemd assumes that your ExecStart command will start the actual service process. It will start that process in its own control group (cgroup), and will monitor it. Any children of the ExecStart process will be members of the same control group.

When the ExecStart process dies, systemd will assume it means your service died. Since you're now using a script to start the actual service, this assumption will be incorrect. At that point, it will kill any processes remaining in the control group to clean up the service (effectively killing your actual service process). Then it attempts to restart the service, and the cycle repeats...

By starting the service indirectly through a script, you effectively changed your service from the simple type to forking type, but did not tell that to systemd. But the forking type has some additional legacy baggage associated with it. It also makes it harder for systemd to monitor your service process; you should avoid it unless absolutely necessary.

Ideally you should keep the start of the actual service process in ExecStart= and have any environment variables specified in the actual .service file with the Environment= option, or in a separate file referred to by EnvironmentFile= option. Any extra start-up commands could become ExecStartPre= and/or ExecStartPost= options; that way you can still keep the default Type=simple and the automatic process monitoring it provides. You can still use ExecStop= with a Type=simple service, if needed.

---

If you use Type=forking, systemd will still track the service through its control group. If the service creates other processes, it won't know which of them is the main process of the service, and so you'll need to provide at least the PIDFile= option to help systemd kill the main service process first when stopping the service, or a suitable ExecStop= that does something more service-friendly than just blindly killing processes.

If the service's control group becomes empty of processes, systemd will still detect the service has failed. But with Type=forking and no PIDFile=, the main process of the service might die and the failure might remain undetected as long as at least one of its child processes remains.

When the ExecStop= process is done, if there are any processes left in the service's process group, systemd assumes it means the orderly shutdown failed for some reason, and will immediately use SIGKILL to clean up the remaining processes in the control group unless specified otherwise using the various options listed in the systemd.kill(5) man page.

So, if you use Type=simple, you won't need a PIDFile= and don't have to worry about what to do if a crash of the service or the entire system leaves you with a stale PID file.

If you use Type=forking and your service uses more than one process, you should use a PIDFile= so that systemd can correctly identify the master process of your service, both for monitoring purposes and for killing when necessary.

If your service needs a more elaborate shutdown process than just "send a SIGTERM to its main/only process", use ExecStop= regardless of the Type= option, but be aware that it needs to handle any necessary wait/timeout too; the expected result is that the service shutdown is completed when the ExecStop= process ends. If there are any remaining processes in the service's control group after that, systemd will assume they can safely be killed and will immediately use SIGKILL to clean them up.