Why doesn't SIGKILL terminate a stopped program (yes)?

Question

I'm using Ubuntu 14.04 and I'm experiencing this behavior I can't seem to understand:

1. Run the yes command (in the default shell: Bash)
2. Type CtrlZ to stop yes
3. Run jobs. Output:
   [1]+ Stopped yes
4. Run kill -9 %1 to stop yes. Output:
   [1]+ Stopped yes
5. Run jobs. Output:
   [1]+ Stopped yes

This is on Ubuntu 3.16.0-30-generic running in a parallels virtual machine.

Why didn't my kill -9 command terminate the yes command? I thought SIGKILL can't be caught or ignored? And how can I terminate the yes command?

Answer 1

Signals are blocked for suspended processes. In a terminal:

$ yes
...
y
y
^Zy

[1]+  Stopped                 yes

In a second terminal:

$ killall yes

In the first terminal:

$ jobs
[1]+  Stopped                 yes

$ fg
yes
Terminated

However SIGKILL can't be blocked. Doing the same thing with killall -9 yes from the second terminal immediately gives this in the yes terminal:

[1]+  Killed                  yes

Consequently if kill -9 %1 doesn't terminate the process right away then either bash isn't actually sending the signal until you fg the process, or you have uncovered a bug in kernel.

Answer 2

Don't panic.

There's nothing funky going on. There's no kernel bug here. This is perfectly normal behaviour from the Bourne Again shell and a multitasking operating system.

The thing to remember is that a process kills itself, even in response to SIGKILL. What's happening here is that the Bourne Again shell is getting around to things before the process that it just told to kill itself gets around to killing itself.

Consider what happens from the point where yes has been stopped with SIGTSTP and you've just executed the kill command with the Bourne Again shell:

1. The shell sends SIGKILL to the yes process.
2. In parallel:
   1. The yes process is scheduled to run and immediately kills itself.
   2. The Bourne Again shell continues, issuing another prompt.

The reason that you are seeing one thing and other people are seeing another is a simple race between two ready to run processes, the winner of which is entirely down to things that vary both from machine to machine and over time. System load makes a difference, as does the fact that your CPU is virtual.

In the interesting case, the detail of step #2 is this:

1. The Bourne Again shell continues.
2. As part of the internals of the built-in kill command it marks the entry in its job table as needing a notification message printed at the next available point.
3. It finishes the kill command, and just before printing the prompt again checks to see whether it should print notification messages about any jobs.
4. The yes process hasn't had the chance to kill itself yet, so as far as the shell is concerned the job is still in the stopped state. So the shell prints a "Stopped" job status line for that job, and resets its notification pending flag.
5. The yes process gets scheduled and kills itself.
6. The kernel informs the shell, which is busy running its command line editor, that the process has killed itself. The shell notes the change in status and flags the job as notification pending again.
7. Simply pressing enter to cycle through the prompt printing again gives the shell the chance to print the new job status.

The important points are:

• Processes kill themselves. SIGKILL isn't magical. Processes check for pending signals when returning to application mode from kernel mode, which happens at the ends of page faults, (non-nested) interrupts, and system calls. The only special thing is that the kernel doesn't allow the action in response to SIGKILL to be anything other than immediate and unconditional suicide, with no return to application mode. Importantly, processes need to both be making kernel-to-application-mode transitions and be scheduled to run in order to respond to signals.
• A virtual CPU is just a thread on a host operating system. There's no guarantee that the host has scheduled the virtual CPU to run. Host operating systems are not magical, either.
• Notification messages aren't printed when the job state changes happen (unless you use set -o notify). They are printed when next the shell reaches a point in its execution cycle that it checks to see whether any notifications are pending.
• The notification pending flag is being set twice, once by kill and once by the SIGCHLD signal handler. This means that one can see two messages if the shell is running ahead of the yes process being rescheduled to kill itself; one a "Stopped" message and one a "Killed" message.
• Obviously, the /bin/kill program doesn't have any access to the shell's internal jobs table; so you won't see such behaviour with /bin/kill. The notification pending flag is only set the once, by the SIGCHLD handler.
• For the same reason, you won't see this behaviour if you kill the yes process from another shell.

Answer 3

What you're observing is a bug in this version of bash.

kill -9 %1 does kill the job immediately. You can observe that with ps. You can trace the bash process to see when the kill system call is called, and trace the subprocess to see when it receives and processes the signals. More interstingly, you can go and see what's happening to the process.

bash-4.3$ sleep 9999
^Z
[1]+  Stopped                 sleep 9999
bash-4.3$ kill -9 %1

[1]+  Stopped                 sleep 9999
bash-4.3$ jobs
[1]+  Stopped                 sleep 9999
bash-4.3$ jobs -l
[1]+  3083 Stopped                 sleep 9999
bash-4.3$ 

In another terminal:

% ps 3083
  PID TTY      STAT   TIME COMMAND
 3083 pts/4    Z      0:00 [sleep] <defunct>

The subprocess is a zombie. It's dead: all that's left of it is an entry in the process table (but no memory, code, open files, etc.). The entry is left around until its parent takes notice and retrieves its exit status by calling the wait system call or one of its siblings.

An interactive shell is supposed to check for dead children and reap them before printing a prompt (unless configured otherwise). This version of bash fails to do it in some circumstances:

bash-4.3$ jobs -l
[1]+  3083 Stopped                 sleep 9999
bash-4.3$ true
bash-4.3$ /bin/true
[1]+  Killed                  sleep 9999

You might expect bash to report “Killed” as soon as it's printing the prompt after the kill command, but that isn't guaranteed, because there's a race condition. Signals are delivered asynchronously: the kill system call returns as soon as the kernel has figured out which process(es) to deliver the signal to, without waiting for it to be actually delivered. It's possible, and it does happen in practice, that bash has time to check on the status of its subprocess, find that it's still not dead (wait4 doesn't report any child death), and print that the process is still stopped. What is wrong is that before the next prompt, the signal has been delivered (ps reports that the process is dead), yet bash still hasn't called wait4 (we can see that not only because it still reports the job as “Stopped”, but because the zombie is still present in the process table). In fact, bash only reaps the zombie the next time it needs to call wait4, when it's run some other external command.

The bug is intermittent and I couldn't reproduce it while bash is traced (presumably because it's a race condition where bash needs to react fast). If the signal is delivered before bash checks, everything happens as expected.

Answer 4

Something funky may be happening on your system, on mine your recipe works nicely both with and without the -9:

> yes
...
^Z
[1]+  Stopped                 yes
> jobs
[1]+  Stopped                 yes
> kill %1
[1]+  Killed                  yes
> jobs
> 

Get the pid with jobs -p and try to kill it as root.