How can I debug random Emacs server crashes?

Question

I generally have a long-running instance of emacs --daemon, which I connect to with various emacsclients. Recently I have started experiencing random crashes. I say "random" because I haven't been able to figure out what causes the crashes. Often it happens e.g. when I leave my work computer over the weekend. However, I also remember at least one time when I started Emacs again after a crash, and it crashed again after a couple hours, during the same session.

I started an instance with emacs -Q (and then did M-x server-start) on my unused laptop, and it has not crashed yet. (Currently it has been: about a week.) One day I had need to use Emacs on that laptop, so I started up a separate session by doing emacs from a terminal window, and did not start the server. I left that session running, and when I came back a day or two later, I found that it had crashed, giving Segmentation fault. Thus, it would seem that this is not related to running Emacs in daemon mode.

I would do a binary search of my init file, but due to the randomness of the crashes, it would be a pain to go without much of the functionality I'm used to for an extended amount of time. Also, I wouldn't have a good way to know when I had fixed it, and the crashes weren't occuring.

I use Emacs 24.4, on Debian GNU/Linux 8, and on Ubuntu. (I have used Emacs on Windows before, but didn't use the --daemon, and didn't experience any crashes.)

How else can I debug these random crashes?

Edit: I get the same result when I launch Emacs and then do M-x server-start, and also when I use the Lucid toolkit.

I ran gdb emacs, and then at the (gdb) prompt did run, and this was the output:

Starting program: /usr/bin/emacs
[Thread debugging using libthread_db_enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Warning: Cannot convert string "-*-courier-medium-r-*-*-*-120-*-*-*-*-io8859-*" to type FontStruct

(Note that the last line also appears as terminal output when I launch Emacs outside of GDB.)

In Emacs, I did M-x server-start, and started work. It didn't crash all day, so I left it running over the weekend. When I got back today, I found Emacs unresponsive, and this additional output in GDB:

Program received signal SIGSEGV, Segmentation fault.
mark_object (arg=42366066) at alloc.c:5601
5601    alloc.c: No such file or directory.

After that line, control returned to the GDB prompt.

Edit 2: After building Emacs from source, I again ran it from GDB and did M-x server-start. Today I again found it unresponsive, and this was the output in GDB:

Starting program: [snip]/emacs-24.5/src/emacs 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffee188700 (LWP 20124)]
[New Thread 0x7fffe7df2700 (LWP 20125)]
[New Thread 0x7fffe75f1700 (LWP 20126)]

Program received signal SIGSEGV, Segmentation fault.
mark_object (arg=47086818) at alloc.c:5973
5973    {

Answer 1

The way to debug crashes is to capture the crash in a debugger. The debugger gives you the tools you need to diagnose why the crash happened.

Specifically you could do is run the emacs server inside gdb. That is, instead of running it directly, run gdb emacs --daemon instead. Then, when it crashes it be kept alive by the debugger and you can diagnose the problem. You report doing this, but not whether you collected any information about the crash. For example, you didn't collect the stack, or information about local variables, etc. On the other hand, it looks like it crashed during garbage collection, which means that the real bug might have occurred much earlier. Corrupted or overwritten data structures can cause all kinds of bad behavior, including crashes, later on when a program tries to use those data structures. Thus it's not always easy to find out exactly what went wrong or how to fix it. The other disadvantage is that if you exit gdb, you lose all information about the crash.

Another option that wasn't available in 2015 when this question was originally asked is to record the emacs server with rr (https://rr-project.org/). Once it crashes you can replay the saved recording as often as necessary. You can set a data watchpoint on the corrupted data structure that caused the crash, and run emacs in reverse until something writes to that location. This frequently pinpoints the cause of the corruption; you'll be stopped right at it. Other times you might have to follow the chain of evidence further back to find the real problem. You can even pack up the recording along with all the other files necessary to replay it and send that recording to another developer who is better at debugging things, if you can't figure it out yourself.

This specific crash has probably been fixed some time in last five years, but there are probably other bugs waiting to be found and erradicated.