13

I am connected to a beagleboard via tty and I suddenly see this message :

Broadcast message from root@arm
        (unknown) at 11:40 ...

The system is going down for reboot NOW!
[  143.036193] Restarting system.

The system is Ubuntu 12.10. Is there a place where I can find why the system decided to go down ?

UPDATE: More info

  • This actually happens all the time, the board reboots every 1 to 5 minutes.
  • I made sure no program or daemon runs at startup.
  • I can't find Restarting anywhere in any file in /var/log
  • In syslog before a reboot there is consistently a Feb 10 09:20:00 arm rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="317" x-info="http://www.rsyslog.com"] exiting on signal 15.
Thomas
  • 893
  • @JMCF125 I have updated my question with some more info – Thomas Feb 10 '14 at 11:05
  • It turns out that rsyslogd is simply the daemon that gives those messages... Perhaps you got a rare Linux malware... 1 to 5 minutes is little time, try to insert a Live CD and find out more. If you get nothing, I'd advise you to reinstall, possibly another distro. – JMCF125 Feb 10 '14 at 12:05
  • I was at the beginning thinking to a physical problem on the board like electrical voltage going a little too low, but given would the system know about it ? – Thomas Feb 10 '14 at 12:39
  • If the power was the problem, with the Live CD the same would happen. – JMCF125 Feb 10 '14 at 19:20

3 Answers3

12

The way I'd track this down is to replace the utilities halt and shutdown with a script.

First create a script such as the following at /bin/fakehalt:

#!/bin/bash
exec >>/tmp/fakehalt.log 2>&1
date
echo "CMD=$0 PID=$$"
ps -ef --forest
echo '========'

Then install it with:

chmod a+x /bin/fakehalt
mv /sbin/halt /sbin/halt.orig
ln -s /bin/fakehalt /sbin/halt
mv /sbin/shutdown /sbin/shutdown.orig
ln -s /bin/fakehalt /sbin/shutdown

This will create a log file at /tmp/fakehalt.log each time it is called. It'll log the name it was called as (halt or shutdown) it's own PID, and then a tree diagram of all processes at the time.

This should give you all the necessary info to track it down. Just look through the ps tree and find what called the script.

/sbin/reboot should be a symlink to /sbin/halt. If it isn't, then replace it as well.

If this still doesn't capture it, replace /sbin/init as well (as it can also be used to reboot the system). But this is dangerous as if the system reboots, it won't come up properly.

phemmer
  • 71,831
  • I like your solution, I have tried it and it helped me find the problem (a job in fcron left by someone else that did shutdown on purpose when a check was false) – Thomas Feb 10 '14 at 14:17
  • 1
    @Thomas You mentioned earlier that fcron was empty, just wondered why the task didn't show, can you see it now ? – X Tian Feb 10 '14 at 19:30
  • @XTian human error, I checked for cron but not fcron – Thomas Feb 11 '14 at 07:17
  • @Thomas, BTW, why was there a job in fcron to shut down? – JMCF125 Feb 14 '14 at 12:49
  • @JMCF125 if something that should be plugged in does not show up in /dev/... we reboot for instance – Thomas Feb 15 '14 at 07:53
  • @Thomas, was that the case? What wasn't plugged in? Can you show the job? – JMCF125 Feb 15 '14 at 12:01
  • 1
    @JMCF125 Yes it was the case (beagleboard usb disconnect issues) and led me to read very long google groups threads. It was my mistake with the fcron, this was a false fire. Somehow the script in fcron says "reboot" and that's why there is a broadast message in kern.log. Thanks for following up – Thomas Feb 15 '14 at 12:34
  • @Thomas, you're welcome. This was an interesting problem. But I still don't see why you didn't mark this answer as accepted, if it is the one you used. – JMCF125 Feb 15 '14 at 13:02
3

Run you kernel with something like init=/bin/bash. This should make that no processes are running than the this shell.

Verify that there is on rebooting issue anymore.

Next find out what init system you use normally and have a look at all services started by it.

Try to disable services until the rebooting issue disappears.

michas
  • 21,510
  • in what file do you write init=/bin/bash ? – Thomas Feb 10 '14 at 14:10
  • This depends on your boot loader. (grub, syslinux, etc.) Usually your boot loader should give you some interactive way to choose your kernel and add additional parameters. It also has some configuration file (usually in /boot) to permanently add parameters. – michas Feb 10 '14 at 14:39
2

Well it's one of these possibilities.

  • Someone is running shutdown
  • cron is running shutdown (or has started something which is running shutdown)
  • power management triggers a shutdown (no power, hibernate, sleep or idle system, the power hardware button is pressed)
  • reboot requested after package install
X Tian
  • 10,463
  • Unfortunately cron and fcron are empty, and I am the only one connected to the board, no one is running shutdown – Thomas Feb 10 '14 at 11:20
  • The OP says: «I made sure no program or daemon runs at startup.». And I doubt someone has hacked into his computer to run shut it down once in a while. Maybe it's something else. – JMCF125 Feb 10 '14 at 11:20
  • Search for shut (case independent not restarting) in var/log/* – X Tian Feb 10 '14 at 11:52
  • no result for "shut" – Thomas Feb 10 '14 at 12:37