33

I foolishly started a job that turned out to be so big and busy that it froze everything. I wish I could type a kill command or use xkill, but the system is unresponsive, apart from the audible swapping. On Windows, Ctrl+Alt+Del helps in these situations; does Linux has a way to knock through into an overloaded system?

Just saw this one and couldn't stop myself from sharing:

xkcd: Chorded Keyboard

Edgar Magallon
  • 4,820
Michael
  • 765

5 Answers5

51

Ctrl + Alt + F4 opens a console window, where you can login and kill stuff as necessary or reboot the system. Use Ctrl + Alt + F2 or Ctrl + Alt + F1 to go back.

In some cases you can restart the gnome session by pressing Alt + F2, and the R in the window that opens. This should leave all programs running, but gnome itself will restart, so if the issue is in gnome it may help.

If the above don't help, you can do a warm reboot, by pressing the following key sequence:

While keeping pressed down both the Alt and Print Screen keys, sequentially (one by one) press the keys:

R E I S U B

This will sync and unmount the file system and do a safe reboot. The keys have the following meaning:

  • R: Switch the keyboard from raw mode to XLATE mode
  • E: Send the SIGTERM signal to all processes except init
  • I: Send the SIGKILL signal to all processes except init
  • S: Sync all mounted filesystems
  • U: Remount all mounted filesystems in read-only mode
  • B: Immediately reboot the system, without unmounting partitions or syncing

Source

Finally, if all else fails, keep the power-on button pressed for a few seconds to force a cold reboot, or take the power cable/battery out ;-).

Cyrille
  • 113
user000001
  • 3,635
  • 3
    I use it so rarely I forgot about the Magic SysRq incantations. – DopeGhoti Feb 15 '22 at 19:57
  • 1
    @DopeGhoti: Yes it's rarely needed, linux is generally very stable unless you run out of virtual memory or disk space. I learned the sequence by heart when I had a defective SSD drive, and the system kept freezing on me. – user000001 Feb 15 '22 at 20:03
  • 12
    If resorting to Sysrq, I like to try F, which invokes the OOM killer, before sending any other signal to all processes. Many times I've recovered from a stray, memory hogging process without losing my current session. – Quasímodo Feb 15 '22 at 20:05
  • @Quasímodo: That's really cool, I didn't know such a thing existed, nice! – user000001 Feb 15 '22 at 20:07
  • 5
    There are a lot more functions than REISUB and F: https://en.wikipedia.org/wiki/Magic_SysRq_key#Commands – DopeGhoti Feb 15 '22 at 20:08
  • 1
    Which keymap is in effect for Magic SysReq? Is it the currently loaded one for the virtual terminal? – Toby Speight Feb 16 '22 at 09:15
  • 1
    @Quasímodo In Debian, you have to enable that beforehand with echo 1 | sudo tee /proc/sys/kernel/sysrq (enables all Magic SysRq keys; other numbers enable certain keeys and disable others). There's also a config file to do this permanently; I think /etc/sysctl.conf. – wizzwizz4 Feb 16 '22 at 10:06
  • 11
    "While keeping pressed down both the Alt and Print Screen keys, ..." - You don't need to hold down Print Screen while doing the other keys! Just hold down Alt, and then press, in sequence, Print Screen, R, E, etc. Much easier. Also consider SysRq+K, which will kill all processes on the current virtual terminal. You'd lose your graphical session, including hopefully your runaway process, but not the system. (I'm not 100% sure SysRq+K still works as expected with Wayland, never tried it) – marcelm Feb 16 '22 at 10:48
  • @marcelm indeed, holding both Alt and PrintScreen while typing REISUB may not even work in the end, see these Q&A. – Ruslan Feb 16 '22 at 13:02
  • 12
    "Reboot Even If System Utterly Broken" – Drake P Feb 16 '22 at 16:38
  • 3
    @DrakeP I've also heard "Raising Elephants Is So Utterly Boring". – Soren Bjornstad Feb 16 '22 at 20:05
  • Maybe add the warning to wait quite a few between each key press (if you press S, U and B too close together the system will probably not have time to properly sync, nor unmount, before it is forceably shut down) (and wait >15s between E and I as well, for the same reason) – Olivier Dulac Feb 17 '22 at 10:55
  • Maybe you can add, that one does not need the full REISUB sequence in all cases. The idea of R is for example to get access to the keyboard when something blocked it by switching to a mode that does not allow you to input the commands needed to unblock it again. And K often works for killing an X11 session (when not even ctrl-alt-backspace is working anymore) and afterward you can login again. – allo Feb 17 '22 at 17:45
10

Depending on your installation and environment, you may be able to kill or restart the entire XOrg session with Ctrl-AltBackspace, but some modern distributions have been disabling that shortcut by default, unfortunately.

If your virtual terminal service is still working (typically getty or something similar), you may be able to switch to an alternate local console with e. g. LAlt-F1, and log in there to issue kill commands at a CLI shell if the system is not so bogged down that you cannot even log in.

If you are on a network and your host has a running secure shell daemon, you might also be able to log in through ssh to kill your processes that way.

DopeGhoti
  • 76,081
  • 1
    I wouldn't call setting DontZap unfortunate, since that keystroke is wired into my fingers as backward-kill-sexp. It's horrendous when simply deleting some code kills the entire X session. – Toby Speight Feb 16 '22 at 07:56
  • 2
    @TobySpeight Not to mention that it is a way to bypass many screenlockers, since they are GUI, and then leave the intruder with one's shell on the virtual console. – Quasímodo Feb 16 '22 at 12:50
  • 1
    Not generally true when it is enabled. If you C-A-Bksp on a system using, say, gdm, XOrg will restart and deposit you at the login dialog. It will be true if you started the GUI session yourself from a console login however. Unless your shell was startx or something. 'Unfortunate' here is, as it usually is, in the eye of the beholder (: – DopeGhoti Feb 16 '22 at 13:57
  • @Quasímodo: Ctrl+Alt+Fn bypasses al the screen lockers I tried so that's no issue. – Joshua Feb 16 '22 at 17:36
  • It doesn't bypass it per se; the screen is still locked. You're just selecting a different (virtual) screen. – DopeGhoti Feb 16 '22 at 17:38
  • 2
    You can run exec startx to have the shell terminate when startx terminates. – allo Feb 17 '22 at 17:47
10

For the specific case of swapping, what you want to do is to invoke the out-of-memory killer.

This can be done with the SysRq key combination Alt+SysRq+F. It will stop the process that is most heavily allocating RAM.

However, the key combination is by default disabled on many systems such as Ubuntu because it can potentially enable to circumvent screen lock. It can be enabled by editing /etc/sysctl.d/10-magic-sysrq.conf.

jpa
  • 1,269
3

You can have your init daemon invoke any (previously configured) program of your choosing by pressing Alt+↑ at a console, which will send SIGWINCH to process 1. The keyboard must not be in raw mode, which means this will not work under an X session without pressing Alt+SysRq+R first (which will work assuming the magic SysRq key has been enabled).

In a systemd-based system, the action to take is configured by defining a unit called kbrequest.target; see systemd.special(7).

Under sysvinit, this is configured by an /etc/inittab entry starting with ::kbrequest:; see inittab(5).

Other init systems may have different ways to configure how they react to this key combination.

user3840170
  • 1,832
3

Im a little surprised that no-one has mentioned the problem, rather than the fix. The issue is that your machine shouldn't (ideally) be unresponsive when under memory pressure. This is a problem with the current swap strategy, and has come up in kernel discussions previously, as the "elephant in the room".

There are user-space "fixes" which help mitigate the problem. I recommend something like earlyoom, which will kill processes before you run out of ram, preventing the swap problem in the first place.

Earlyoomrecommender
  • 31