How to solve this memory issue gracefully?

Question

I have a standard Linux (Debian testing) laptop, with a swap partition.

I do a lot of experiments with it. Some of them are really memory hungry and the way Linux behaves by default is an issue for me... Let's give a stupid example:

Sit in front of the laptop
Open a terminal
Type python, then a = [0]*100000000

Now chances are high that you won't have enough RAM to handle that big list. Linux will fill the RAM, then the swap and, a couple of minutes later, the OOM killer will be triggered off and kill (almost) random services and hopefully, if you hit Ctrl+C at the good time, python, and if the terminal still had focus, the computer will become responsive again.

I'd like to enforce some memory limits to avoid that unwanted swapping and to refuse to a process the right to allocate more memory than I have (in RAM). If the memory demand is below a certain limit or asked by root, then just kill the most memory hungry process of any user except root.

ulimit -Sv [mem] I hear in the back!

Ho Ho! "Use cgroups via cgexec!" someone says at the first row!

Yes, you are right: these are indeed very good solutions. But:

They do not apply system-wide
The limits are set per-process
The limits are static, disregarding the real amount a free RAM (AFAIK)
Here and there, they say these are not really a good solution to enforce hard limits.

What I'd like is that the kernel say: "You belongs to user foo (not root), you use a lot of memory and we're gonna run out of memory. Sorry dude... die now!"

Or: "What the hell are you doing? You need x MB and there is only y MB available. Yes, SWAP is empty, but you don't intend to use the SWAP to do your dirty work, do you? No, I said no! No memory for you! If you insist, you're gonna die!"

There is already a algorithm described in this article which helps the OOM killer to choose the correct process. Changing /proc/sys/vm/overcommit_memory affects the kernel behaviour on low memory. — jofel, Feb 12 '13 at 13:00
Yes, but the overcommit_memory special file uses RAM+SWAP as usable memory. I'm still gonna swap :) — , Feb 12 '13 at 13:03
You also need to explain how this is not a duplicate of this: http://unix.stackexchange.com/questions/34334/how-to-create-a-user-with-limited-ram-usage which contradicts you WRT cgroups and individual users. PS. If you don't want to swap, disable swap. — goldilocks, Feb 12 '13 at 13:05
I want swap! I want hibernation, I want unused bytes to be stored away! But I don't want used bytes to be stored there. About the link, ulimits are a bad idea as shown almost everywhere since it is a per process limitation... I fork you know :) About cgroups, this is definitely better but lacks something more general: I'm talking about my laptop but I also own a "calculation" server that we are three to share. If I enforce such per user limits, I'll be limited by the worst case scenario, won't I? — , Feb 12 '13 at 13:19
cgroups do apply to whatever process you decide- put all processes of a user into a separate group and it should do what you want. — peterph, Feb 12 '13 at 13:56
ulimit is for the user. cgroup is for the process and decendants. — vonbrand, Feb 12 '13 at 18:11

score 5 · Accepted Answer · answered Feb 12 '13 at 14:42

Someone suggested in your hear cgroups. Well, try to seek that direction as it can provide you with:

applied to a group of task you choose (thus not system wide but neither per process)
the limits are set for the group
the limits are static
they can enforce hard limit on memory and/or memory+swap

Something like that could bring you closer to your goals:

group limited {
  memory {
    memory.limit_in_bytes = 50M;
    memory.memsw.limit_in_bytes = 50M;
  }
}

This tells that the tasks under this cgroup can use at maximum 50M of memory only and 50M of memory+swap, so when the memory is full, it won't swap, but if the memory is not full and some data could be mapped in swap, this could be allowed.

Here is an excerpt from the cgroup's memory documentation:

By using memsw limit, you can avoid system OOM which can be caused by swap shortage.

Still not exactly what I was expecting. But the difference between what I expect and reality is often quite large :) In this case, I wanted to be sure that I did not miss anything like the overcommit_memory kernel variable. Thank you all. — , Feb 12 '13 at 20:42

score 0 · Answer 2 · edited Apr 18 '19 at 17:01

I run into the same problem frequently. My general workflow involves heavy computation in MATLAB. Occasionally I will inadvertently attempt to allocate a new variable that exceeds the amount of available memory. The system hangs, and I typically have to hard-reboot the machine to get back to work. :P

In my case, and it sounds like in yours also, I was not so much concerned with limiting the amount of memory MATLAB uses to a static amount - I was interested in not having a frozen machine, and I was willing to sacrifice my MATLAB process in order to preserve system responsiveness.

Inspired by a response to this post, I wrote the following script (I called it watch_memory.sh):

#!/bin/bash

MONITOR=$(free | grep 'buffers/cache:')
MEM_USED=$(echo $MONITOR | awk '{ print $3 }')
MEM_FREE=$(echo $MONITOR | awk '{ print $4 }')

MEM_PERC=$(( 100*MEM_USED / (MEM_FREE+MEM_USED) ))

while :; do
    if [ "$MEM_PERC" -gt "95" ]
    then
        kill $1
        echo "$1 killed for using too much memory."
        exit
    fi
    sleep 1

    MONITOR=$(free | grep 'buffers/cache:')
    MEM_USED=$(echo $MONITOR | awk '{ print $3 }')
    MEM_FREE=$(echo $MONITOR | awk '{ print $4 }')
    MEM_PERC=$(( 100*MEM_USED / (MEM_FREE+MEM_USED) ))
done

This script checks every second for the percentage amount of free memory. When the system runs out, your "scapegoat" pid (passed as an argument to the script) gets killed.

Without adjusting the priority (niceness) of the script, it took about 10-20 seconds for the scapegoat to get killed, but it still worked. Running the script with a negative priority resulted in an instant kill after violation (11916 in this example is the pid I want to kill if I run out of memory):

sudo nice -n -5 bash watch_memory.sh 11916

How to solve this memory issue gracefully?

2 Answers2

Linked