28

I am using Debian stretch (systemd). I was running the rsyslog daemon in foreground using /usr/sbin/rsyslogd -n and I did a Ctrl+Z to stop it. The state of the process changed to Tl (stopped, threaded). I issued multiple kill -15 <pid> commands to the process, and the state of the process was the same: Tl. Once I did an fg, it died. I have 3 questions.

  • Why was the SIGSTOP-ed process not responding to SIGTERM? Why does the kernel keeps it in the same state?
  • Why did it get killed the moment it received the SIGCONT signal?
  • If it was because of the previous SIGTERM signal, where was it kept until the process resumed?
nohup
  • 431

2 Answers2

45

SIGSTOP and SIGKILL are two signals that cannot be caught and handled by a process. SIGTSTP is like SIGSTOP except that it can be caught and handled.

The SIGSTOP and SIGTSTP signals stop a process in its tracks, ready for SIGCONT. When you send that process a SIGTERM, the process isn't running and so it cannot run the code to exit.

(There are also SIGTTIN and SIGTTOU, which are signals generated by the TTY layer when a backgrounded job tries to read or write to the terminal. They can be caught but will otherwise stop (suspend) the process, just like SIGTSTP. But I'm now going to ignore those two for the remainder of this answer.)

Your CtrlZ sends the process a SIGTSTP, which appears not to be handled specially in any way by rsyslogd, so it simply suspends the process pending SIGCONT or SIGKILL.

The solution here is also to send SIGCONT after your SIGTERM so that the process can receive and handle the signal.

Example:

sleep 999 &

# Assume we got PID 456 for this process
kill -TSTP 456    # Suspend the process (nicely)
kill -TERM 456    # Terminate the process (nicely). Nothing happens
kill -CONT 456    # Continue the process so it can exit cleanly

The documentation for the GNU C Library explains this quite well, I think (my highlighting):

While a process is stopped, no more signals can be delivered to it until it is continued, except SIGKILL signals and (obviously) SIGCONT signals. The signals are marked as pending, but not delivered until the process is continued. The SIGKILL signal always causes termination of the process and can’t be blocked, handled or ignored. You can ignore SIGCONT, but it always causes the process to be continued anyway if it is stopped. Sending a SIGCONT signal to a process causes any pending stop signals for that process to be discarded. Likewise, any pending SIGCONT signals for a process are discarded when it receives a stop signal

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • Thanks @roaima. Could you please help me understand why, after I issued a kill -15 and then a SIGCONT, it gets killed? Was it because of the kill -15 that I issued before? – nohup Jul 27 '16 at 14:43
  • 1
    @nohup. answer extended, but essentially, "yes; it handled the kill -15 you'd already sent". – Chris Davies Jul 27 '16 at 14:45
  • So is the SIGTERM stored as a pending signal when send to a SIGSTOP'd process, and issued once the process is SIGCONT'd ? – nohup Jul 27 '16 at 14:54
  • 2
    @nohup yes, as per the documentation: « While a process is stopped, no more signals can be delivered to it until it is continued ... The signals are marked as pending, but not delivered until the process is continued. » – Chris Davies Jul 27 '16 at 14:56
  • 1
    See also SIGTTIN and SIGTTOU that also stop processes – Stéphane Chazelas Jul 27 '16 at 15:13
  • 1
    @StéphaneChazelas good point. I've added mention of these but otherwise ignored them. Please feel free to edit as you see fit. – Chris Davies Jul 27 '16 at 16:30
  • 1
    Learning these discrepancies reminds my of my Op Sys class (shudders). Good answer and will probably help a lot of future junior and senior level CS students haha – 8protons Jul 27 '16 at 17:32
  • Truth is you rarely have to deal with this stuff as a user. As a user, send SIGSTOP to pause SIGCONT to continue and SIGKILL to end a process. There are a bunch of reasons why that oversimplification sucks, but it doesn't matter. As a developer you may or may not have to care depending on framework. So while interesting, all of the signals are just an academic excersize. You want to focus on SIGSTOP, SIGCONT, and SIGKILL for day to day stuff. SIGTERM is a nice one to learn too, but not really needed (as a user) – coteyr Jul 28 '16 at 12:27
  • 2
    @coteyr. I disagree: SIGKILL prevents an app from cleaning up, so using SIGTERM is preferable in many (most) cases. – Chris Davies Jul 28 '16 at 13:16
  • 1
    @roaima - good point. But as a user, usually you don't care. It's more "it no worky how do I kill it?" At the point that an average user would throw a SIGTERM or SIGKILL the point is not usually valid. But I will say, that leaning SIGTERM is not a bad thing and as an admin definitely a tool you need in your tool box. – coteyr Jul 28 '16 at 14:21
10

SIGTERM is just like any other signal in that it can be caught by a process. Receiving the signal will just make the process jump to a special signal handler routine. For SIGTERM the default action will be to terminate the process, but e.g. an editor might like to catch the signal so it can save a draft copy of any open files before dying. If the process is stopped, it cannot run the signal handler, but the signal will remain pending until the process continues. Note that the number of signals sent will usually not be saved.

In theory, the system could know if the process has a signal handler installed for SIGTERM, and terminate it immediately if not. But (as per Gilles' comment) POSIX demands that the signal will pend until the process is continued via SIGCONT.

ilkkachu
  • 138,973
  • 4
    Apologies, my previous comment was wrong. While a process is stopped, no signals are delivered to it except SIGKILL and SIGCONT. Even if the signal has its default action which is to kill the process, this is delayed until the process is resumed by a SIGCONT. POSIX mandates this behavior. – Gilles 'SO- stop being evil' Jul 28 '16 at 09:26