1

I learned the other day that Linux really does kill processes if it runs out of memory. I had detached the swap with swapoff command before that happened. Usually, if any process takes up all the memory by a bug, say memory leak in a loop, my system hanged(I know, technically it's just really really slow), not killing the bad process.

How do I prevent this? No software is perfect and I get to run poorly coded programs and encounter this problem every now and then. I have to reset my system every time. Is this just the limitation of modern kernels? I can't just turn off swap because I know the kernel utilises its swap to some extent to utilise physical memory efficiently even if the physical memory is spacious.

Using smaller size of swap might be an option(shorter time before OOM).

2 Answers2

2

The kernel's Out Of Memory Killer is used as a last resort. If you provide enough memory and/or swap for the running processes then it won't be triggered.

As your system runs out of physical memory it will start to use swap space. From this point on it will start to slow down, sometimes dramatically. If you have enough active processes trying to use the same physical memory your system will start to spend more time swapping them in and out than it can actually running them.

When swap is close to being exhausted the kernel will start to kill off processes in an attempt to keep the others running. I haven't ever needed to investigate the criteria the OOMK uses to pick processes so I can't describe them to you here.

The solution is one or more of the following

  • Add more physical memory
  • Add more (or some) swap. This can be a disk partition or file
  • Reduce the number or size of the running processes
  • Fix the code in processes with runaway memory

Some other references

It is possible to set a per-process bound on the amount of memory it can use. When it attempts to exceed this the allocation request fails. Typically this results in a program crashing, either on an uncontrolled fashion because it didn't test the return value of the allocation request, or in a controlled manner with a fatal exception.

Community
  • 1
Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • @Ashethehuman your tone is completely unacceptable. If someone misunderstood your question, then you need to make it clearer. Insulting the people who give their time to answer your questions is, to say the least, counterproductive. We expect our users to be nice, that includes you. – terdon Jan 13 '17 at 09:18
0

This is an old problem that has been bugging me for quite some time. Based on my experience with Linux, its OOM killer is BROKEN. It does NOT work. Chromium browser, it doesn't kill. Other processes, it doesn't kill. This morning the process Tumblerd almost crashed my system using 2.1 GB. Even after 3 minutes wait on a painfully scary frozen system (it survived thanks to my keyboard shortcut below) Linux had NOT killed the process using 2.1 GB, which further proves that it does not work.

I'm not here to ask anyone if it works or not, so save your typing for another person. The usual bla bla bla... (how rude)

The only way I was able to workaround this issue relies on me watching my available memory with free -h all the time, otherwise I might miss the short threshold of time and the whole system becomes frozen, leading to a hard reset. I had set a keyboard shortcut on XFCE for a script that will kill all my Chromium processes, this was sufficient enough for my particular use case. But it would be much nicer if I could find a way to "kill process which is using most memory" (which I'm currently searching at this moment due to the aforementioned freeze earlier this morning)

First, test what process you want to kill using this command:

ps aux | grep STRING
ps aux | grep PORTION_OF_NAME_WITHOUT_QUOTES_OR_SPACES

If it shows the process you want to kill, then put it in the script below inside the parenthesis.

#!/bin/bash
#
kill -9 $(ps aux | grep type=renderer)
# the above will kill all my Chromium processes
# without having to restart my entire browser or losing progress
# incognito window will be kept alive, instead of lost
#
killall tumblerd
# (optional)

Paste that into an empty text file and save it as SOMETHING.SH Then make it executable by running chmod a+x /path/to/SOMETHING.SH and assign it to a keyboard shortcut (it's possible in XFCE)

For the tumblerd process, the best way to avoid problems is removing it from the system, since it's only responsible for generating video thumbnails.

sudo apt remove tumbler

So for anyone that's reading this, if you have the suspicion that Linux' OOM killer is broken, you're not wrong, don't let others guilt you in thinking that you are.

edit: Mistake on my part, avoid using sudo inside of scripts activated by hotkeys, otherwise the rest of the script will not run, since it will not prompt for password, only if ran from terminal directly.

Winampah
  • 139
  • 4
  • I use the Magic SysRq and worked every time for me. I was quite a but young when I posted this. Looking back as a few year older dev, I think it's Windows that cheated. Linux treats all the processes equally when it comes to memory. There's nice for CPU time, but nothing for swapiness. –  Jul 19 '20 at 09:16
  • If I wanted my system to be reliable, I would've used cgroups or ulimit. Systemd supports both, I reckon. –  Jul 19 '20 at 09:18
  • I wouldn't consider the kernel OOM killer "broken". God knows what they did to Windows since we can't really see the source code, but I'm pretty sure Windows kernel treats GUI components differently, whereas display servers on Linux are just regular old userland processes. Linux doesn't discriminate. When it's busy swapping pages, it can't simply give enough CPU time. –  Jul 19 '20 at 09:23