45

My server program received a SIGTERM and stopped (with exit code 0). I am surprised by this, as I am pretty sure that there was plenty of memory for it. Under what conditions does linux (busybox) send a SIGTERM to a process?

codeforester
  • 742
michelemarcon
  • 3,443
  • I can't think of any case when the kernel or a standard tool would send SIGTERM to a random process. What can you tell us about what the program is doing and how it's started? How did you find out about the program's exit status? Can you reproduce the problem? Do you have logs you can check? – Gilles 'SO- stop being evil' Mar 28 '11 at 17:57
  • It is reading and writing to a serial line, and is responding to UDP and TCP requests. I have wrapped the execution on a bash script and therefore I know the exit code. – michelemarcon Mar 29 '11 at 08:14
  • 1
    Posix documentation indicates SIGTERM is strictly a user level event. Is it possible someone else was able to kill your server program? – shellter Mar 29 '11 at 22:48
  • Very unlikely; however, all I know is the return code (0) which is the same that I get when I terminate the process with ^C (SIGTERM). Maybe I'm overlooking something? – michelemarcon Mar 30 '11 at 07:05
  • 4
    You are! The return code 0 means a normal exit. If there was a SIGTERM, $? would be set to 143 (128 + signal number). – Gilles 'SO- stop being evil' Mar 30 '11 at 18:36
  • 1
    Also, ^C is SIGINT, not SIGTERM, and that would exit with a code of 130. – Sparkette Sep 01 '18 at 01:30
  • but why does SIGTERM happen? how do I debug why my python script is exiting with that? – Charlie Parker Mar 01 '21 at 17:27

3 Answers3

38

I'll post this as an answer so that there's some kind of resolution if this turns out to be the issue.

An exit status of 0 means a normal exit from a successful program. An exiting program can choose any integer between 0 and 255 as its exit status. Conventionally, programs use small values. Values 126 and above are used by the shell to report special conditions, so it's best to avoid them.

At the C API level, programs report a 16-bit status¹ that encodes both the program's exit status and the signal that killed it, if any.

In the shell, a command's exit status (saved in $?) conflates the actual exit status of the program and the signal value: if a program is killed by a signal, $? is set to a value greater than 128 (with most shells, this value is 128 plus the signal number; ATT ksh uses 256 + signal number and yash uses 384 + signal number, which avoids the ambiguity, but the other shells haven't followed suit).

In particular, if $? is 0, your program exited normally.

Note that this includes the case of a process that receives SIGTERM, but has a signal handler for it, and eventually exits normally (perhaps as an indirect consequence of the SIGTERM signal, perhaps not).

To answer the question in your title, SIGTERM is never sent automatically by the system. There are a few signals that are sent automatically like SIGHUP when a terminal goes away, SIGSEGV/SIGBUS/SIGILL when a process does things it shouldn't be doing, SIGPIPE when it writes to a broken pipe/socket, etc. And there are a few signals that are sent due to a key press in a terminal, mainly SIGINT for Ctrl+C, SIGQUIT for Ctrl+\ and SIGTSTP for Ctrl+Z, but SIGTERM is not one of those. If a process receives SIGTERM, some other process sent that signal.

¹ roughly speaking

Stéphane Chazelas
  • 544,893
Gilles 'SO- stop being evil'
  • 829,060
  • 2
    Nice explanation of how exit status is determined upon receiving a signal. However, this answer doesn't address OP's question. – codeforester May 23 '17 at 16:29
  • 1
    @codeforester I answered the question in the body, not the question in the title. Well, one of the questions in the body — given that it was based on a misunderstanding, it is a bit messy. I'll add a few words about the rest. – Gilles 'SO- stop being evil' May 23 '17 at 20:43
  • That depends on the shell. In ksh93, it's 256+signum, in yash, it's 384+signum – Stéphane Chazelas May 23 '17 at 21:00
  • Note that using a value above 256 a la ksh is not necessarily better as that prevents it being passed to exit. The yash approach is a good compromise, but see rc for another one. See also Default exit code when process is terminated? – Stéphane Chazelas May 23 '17 at 21:59
  • but why does SIGTERM happen? how do I debug why my python script is exiting with that? – Charlie Parker Mar 01 '21 at 17:25
  • 1
    @CharlieParker SIGTERM happens because some process sends it. It is not sent automatically by the kernel. Something on your system is explicitly calling kill(…, SIGTERM). – Gilles 'SO- stop being evil' Mar 01 '21 at 18:22
  • @Gilles'SO-stopbeingevil' could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess. – Charlie Parker Mar 01 '21 at 18:30
  • 2
    @CharlieParker It's plausible that a HPC cluster would monitor processes and kill those that it considers to have overused resources. You'd have to look at the cluster's documentation or ask the administrators. – Gilles 'SO- stop being evil' Mar 01 '21 at 18:32
24

SIGTERM is the signal that is typically used to administratively terminate a process.

That's not a signal that the kernel would send, but that's the signal a process would typically send to terminate (gracefully) another process.

That's the signal that is sent by default by the kill, pkill, killall... commands.

That's the signal that is sent to daemons to stop them (like upon a service some-service stop), or sent by init before shutdown (followed by SIGKILL for those processes that have not managed to terminate in time upon SIGTERM).

Note that SIGTERM is not the signal that is sent upon ^C. The signal sent upon ^C is SIGINT.

Stéphane Chazelas
  • 544,893
  • but why does SIGTERM happen? how do I debug why my python script is exiting with that? – Charlie Parker Mar 01 '21 at 17:25
  • 2
    @CharlieParker, you'd need to determine what process is sending that SIGTERM to the process running the python interpreter that interprets your script. System logs may have some information. If on Linux, you could use auditd/auditctl to log invocations of kill/tkill/tgkill system calls. – Stéphane Chazelas Mar 01 '21 at 17:30
  • could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess. – Charlie Parker Mar 01 '21 at 18:30
  • 1
    What does 'in time' mean, usually? How much time is there between SIGTERM and SIGKILL on shutdown? Milliseconds? Seconds? Minutes? – Silicomancer May 14 '21 at 21:54
1

In addition to the other answers, SIGTERM is commonly sent to processes running under a job scheduler in an HPC cluster. The scheduler will often send SIGTERM before it kills the process with SIGKILL.

  • Slurm will send SIGTERM before SIGKILL for jobs that run over the time limit.
  • LSF will send SIGINT then SIGTERM before SIGKILL for jobs that run over the time or memory limit.

Depending on the configured time limit, it may be more or less likely for the process to have completed between SIGTERM and SIGKILL. Defaults fall in the range of tens of seconds.

coyfish
  • 11
  • While we appreciate personal experience, we prefer authoritative references.  Can you link to a document that corroborates this?  Please do not respond in comments; [edit] your answer to make it more complete. – G-Man Says 'Reinstate Monica' Feb 09 '24 at 23:20