Why systemd doesn't know if a service failed?

Question

I am trying to provision spring-boot applications (developed by someone else - I'm not a java programmer) on Ubuntu 18.04. The developer had previously added symlinks in /etc/init.d to enable the service - it starts fine at boot. However if the service subsequently fails, systemd still reports that it is running:

root@example.com:/var/log/apps# systemctl status crm-service
● crm-service.service - LSB: crm-service
   Loaded: loaded (/etc/init.d/crm-service; generated)
   Active: active (exited) since Wed 2020-04-15 12:27:15 BST; 3h 56min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8656 ExecStop=/etc/init.d/crm-service stop (code=exited, status=0/SUCCESS)
  Process: 8703 ExecStart=/etc/init.d/crm-service start (code=exited, status=0/SUCCESS)

Apr 15 12:27:15 example.com systemd[1]: Starting LSB: crm-service...
Apr 15 12:27:15 example.com crm-service[8703]: /var/services/crm-service.conf: line 1: -Xms96M: command not found
Apr 15 12:27:15 example.com crm-service[8703]: Started [8747]
Apr 15 12:27:15 example.com systemd[1]: Started LSB: crm-service.

While I see the same systemd unit file for springboot posted all over the internet, I see nothing there which is likely to resolve this issue.

1. How do I get systemd to see the true status of the service? (it will open a listening socket, but on a random high port)
2. Is there a way to get systemd to try gently to restart a service it knows has failed?

Answer 1

Systemd has many critics and many of it is pretty okay, but it is not. Systemd can track all processes and threads forked (or cloned) from the startup script, and consider the service dead if none of them remained.

The first problem what I can see: systemd does not use start/stop scripts in /etc/init.d, it is only a compatibility addon for that. Instead, systemd uses unit files, i.e. a configuration file for all of its services.

The systemd sysv init compat module effectively generates a unit file for all the services in /etc/init.d. This is not always okay, because the init scripts miss the required information (or it is impossible to extract it from them).

This compat module works that the systemd considers the init script failed, and thus the service not working, if its exit code is non-zero. Zero exit code means successful execution. If the init script is buggy and it gives a zero exit code even on a failure, it will cheat systemd.

The most likely cause of the bug of your init script is that it starts the process in the background, and then exits always with zero. My common experience of most init scripts written by custom providers is that.... maybe most of them have a major place for improvement. Do not trust them, look what they do and fix them. In your case, the best would be to check,

• How it starts your java app
• Where does it start
• As which user does it start

And reproduce the same functionality with a unit file.

There is no way to auto-restart initscripts from systemd, but it is possible from unit files.

Note, if a Java program randomly crashes, it is a serious problem in it, too. All sane java frameworks handle their own fatal errors correctly (they catch all the exceptions, log it and continue).

Another very likely bug in the init script is that it does not find your JVM (most likely: /usr/bin/java), so it substitutes it with an empty string, resulting that it tries to start the JVM flags as a shell command. Obviously there is no -Xms96M command in your system, but an /usr/bin/java -Xms96M ... would work.

An example unit file for a spring boot app:

[Unit]
Description=Crm Spring Boot App Example
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/java -Xms96M ...other flags... your.spring.boot.jar
User=exampleuser
Group=examplegroup
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=exampleapp
WorkingDirectory=/path/to/app/home

[Install]
WantedBy=multi-user.target
Alias=exampleapp.service

This unit file also redirects the standard output and error of the Java process into the syslog.

To autorestart the app, insert

RestartSec=5s
Restart=on-failure

into the [Service] section.

There is a systemd tutorial on GoLinuxCloud.com.