What does systemd exit code EXIT_FDS mean?

Question

I get this message from systemd status after I have stopped my service:

Actice: failed (Result: exit-code) <...> Main PID: 4747 (code=exited, status=202/FDS)

Status FDS is defined in the docs like this:

202 EXIT_FDS Failed to close unwanted file descriptors, or to adjust passed file descriptors.

Starting the service works fine, no errors are reported by systemd status

Questions

1. What does EXIT_FDS mean in more practical detail?
2. Is the status code from my application, or from systemd itself?
3. My application opens a TCP socket, which it doesn't close when stopped. Is that the reason?
4. If so, can I make systemd ignore the lingering socket and not report it as an error?

Details

The full status message:

tool-user@tool-box:~$ systemctl status tool.service
● tool.service - Tool application
   Loaded: loaded (/home/tool-user/tool.service; linked; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2022-02-07 14:14:46 CET; 3s ago
  Process: 4758 ExecStop=/bin/bash -c tool-stop && while ps -p $MAINPID >/dev/null
  Process: 4601 ExecStart=/bin/bash -c tool-start (code=exited, status=0/SUCCESS)
 Main PID: 4747 (code=exited, status=202/FDS)
Feb 07 14:14:31 tool-box systemd[1]: Starting Tool application...
Feb 07 14:14:32 tool-box bash[4601]: Server started on port 44680
Feb 07 14:14:32 tool-box systemd[1]: Started Tool application.
Feb 07 14:14:44 tool-box systemd[1]: Stopping Tool application...
Feb 07 14:14:45 tool-box systemd[1]: tool.service: Main process exited, code=exited, status=202/FDS
Feb 07 14:14:46 tool-box systemd[1]: tool.service: Failed with result 'exit-code'.
Feb 07 14:14:46 tool-box systemd[1]: Stopped Tool application.

The service definition file looks like this:

[Unit]
Description=Tool application
# Standard dependencies for web server
After=network.target remote-fs.target nss-lookup.target httpd-init.service
[Service]
Type=forking
Restart=on-failure
RestartSec=10
ExecStart=/bin/bash -c 'toolStart'
ExecStop=/bin/bash -c 'toolStop && while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done'
User=tool-user
StandardOutput=syslog
StandardError=syslog
TimeoutStopSec=60
[Install]
WantedBy=multi-user.target

OS: Ubuntu 18.04 Server, run in VirtualBox on Windows 10.

tool-user@tool-box:~$ uname -a
Linux tool-box 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

Since the service has Type=forking, the ExecStart process had PID 4758 and the exit code you're asking about is listed with main PID 4747, we can conclude that systemd managed to fork() a child process which then successfully execve()'d the ExecStart process, and so the table of systemd-specific exit codes does not apply here.

The systemd-specific table of exit codes would apply if the error was from the actual systemd child process after the fork() but before the execve(): specifically, error 202 would mean e.g. a problem in implementing the StandardInput=, StandardOutput= or StandardError= directives in the service definition. But since the ExecStart is specifically reported to have been PID 4601 and having exited with status=0/SUCCESS, that was not what happened here. The ExecStop was executed as PID 4758, so it's not from that one either.

The status code 202 is from the "main process" of your application (the one that had PID 4747), and it means whatever the application developer wanted it to mean.

The lingering TCP socket is not the cause: since your application process died, the kernel will have cleaned up any lingering sockets it may have had.

Of course, if the application did not use the SO_REUSEADDR socket option, it might not be possible to immediately restart the application and have it use the same port number, until the lingering socket's TIME_WAIT has expired... but that's not systemd's problem; that's something the application will have to deal with on its own.

The /FDS part comes from the exit_status_to_string() function in shared/exit-status.c file in the systemd source code package.

That function is supposed to add a brief hint to what the status code may mean, if the code has any standardized meaning. The function can take a parameter that determines which set(s) of status code hints to use, but when systemctl status uses the function (i.e. in file systemctl/systemctl-show.c, it (as of this writing) seems to always call it with that parameter set to EXIT_STATUS_LIBC | EXIT_STATUS_SYSTEMD, i.e. "show the status code hints according to the usage of libc and systemd itself" without checking if the status code in fact came from a process that was a member of the systemd software suite or not.

The end result is that status 202 always gets /FDS appended to it, whether it's known to have the systemd-specific meaning "Failed to close unwanted file descriptors, or to adjust passed file descriptors" or not. It's just a simple table lookup: don't presume it has any more intelligence than that.

(In Unix programming literature and programmer jargon, "fds" is a pretty universal shorthand for the words "file descriptors". The /FDS also suggests the symbolic name of status code 202 in systemd's code: EXIT_FDS - and since all systemd's status code symbols have the EXIT_ prefix, it's chopped of for brevity.)