On Linux, how can I track what prevents a reboot of the system after issuing it via sudo reboot?
I always thought that this caused switching the runlevel and in the end the reboot, but now I was stumped when several times the machines hadn't rebooted after being told to.
I ran into this problem once when my system was running uninterruptible processes. We went into a very nasty situation, where our Apache processes were seemingly waiting for something from the disk, and couldn't be killed.
As far as I know, when shutting down, Linux sends SIGTERMs to its processes, and a SIGKILL a few seconds later for those who dared disobey. However, the kernel itself will protect these processes if they are in uninterruptible state: this helps preventing data loss when the hanging system call involves an I/O operation. Right now, I can't really think of anything else capable of discarding* a SIGKILL, so other elements of answer could be brought here.
* this is a rather simplified (if not erroneous) way to see it, and it does not take signal asynchronicity into account. See Gilles' comment for more details on this point.
However, when it comes to uninterruptible processes...
How to kill an uninterruptible process without rebooting: somehow make the system call terminate. Frequently the most effective manner to do this without hitting the power switch is to pull the power cord. You can also become a kernel hacker and make the driver use TASK_KILLABLE, as explained in the LWN article.
You can get a list of all running uninterruptible processes with...
$ ps -eo 'state,pid,args' | awk '$1 == "D" {print;}'
$ ps -eo 'state,pid,args' | grep -E '^D'
$ ps -eo 'state,pid,args' | egrep '^D'
...
