How to limit resource usage to save CPU+RAM for a certain process?

Question

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is to tell the machine: "Do what ever you want, but leave out 20MB and some CPU for sshd!".

How can that be done?

Sshd should not be blowing up like that, it's normally a very well behaving daemon. Make sure it's updated to the latest/greatest. What are you doing with it? Are you piping massive data through it? — Marcin, Jun 07 '11 at 12:16
@Marcin I don't think I'm doing anything big, text editing. Even if sshd is well behaved, I think it would stuck if there is not enough CPU/RAM to use, wouldn't it? — phunehehe, Jun 07 '11 at 15:08
When linux runs out of memory, it starts killing off processes to reclaim memory. You might be seeing OOM in action (out of memory killer). I don't know why would it decide to kill your sshd consistently though. Have you ever seen sshd actually leak (grow in size) horribly, before it blows up, or does it work fine one moment and is dead the next? — Marcin, Jun 07 '11 at 21:30
@Marcin chances are if he's OOM the system is bogged down so much he sshd just can't respond, it's not an sshd problem but an issue with his load. Unfortunately I don't know the answer to this. — xenoterracide, Jun 08 '11 at 12:59
@phunehehe are you sure that the problem is no memory for ssh and not your load is through the roof? does sshd give any errors? what is the load on the box? and what is causing the system to run out of memory. — xenoterracide, Jun 08 '11 at 13:03
@xenoterracide I'm not really sure. I guess it lacks either memory or CPU power. Normally the memory stays at 80-90% or it's very possible to blow up that way. Most of the load is from Apache (we host a bunch of webapps with many different frameworks). — phunehehe, Jun 08 '11 at 13:39

score 9 · Answer 1 · answered Jun 07 '11 at 10:14

9

You can probably achieve something like that by using cgroups with the Memory resource controller.

I guess you'd put all your resource-consuming tasks in a limited (CPU & RAM) cgroup, and leave sshd "outside" so that it isn't restricted.

(Adding more swap, even in the form of a swap file, might be a good option though.)

answered Jun 07 '11 at 10:14

Mat

52,586

I'm still hesitant, it doesn't look easy (spoiled by Ubuntu). – phunehehe Jun 07 '11 at 15:43
oh it's certainly not easy :-) Adding swap is by far easier. – Mat Jun 07 '11 at 15:44
@phunehehe welcome to being a Sys Admin, it's not easy. But now you have to learn how to do not easy stuff. – xenoterracide Jun 08 '11 at 13:00

score 6 · Answer 2 · answered Jun 07 '12 at 18:47

Oh but cgroups are easy :) Install the libcgroup package. Create a /etc/cgconfig.conf:

mount {
    cpu     = /cgroup/cpu_and_mem;
    cpuacct = /cgroup/cpu_and_mem;
    memory  = /cgroup/cpu_and_mem;
}

group sshd {
        cpu {
                cpu.shares="500";
        }
        cpuacct {
                cpuacct.usage="0";
        }
        memory {
                memory.limit_in_bytes="1G";
        }
}

group nosshd {
        cpu {
                cpu.shares="500";
        }
        cpuacct {
                cpuacct.usage="0";
        }
        memory {
                memory.limit_in_bytes="1G";
        }
}

Start the cgconfig process which will create the hierarchy, cgroups, and set the limits. If that succeeds, you have two cgroups, both of which have 50% of the CPU assigned and 1G of memory available (don't know what your actual amount of available memory is; assuming it's 2G in this example). Now you just need to move all the tasks (ie all processes running on the system) from the root group into the nosshd cgroup:

cgroup]# cat tasks >> nosshd/tasks
cgroup]# echo > tasks

Then you just need to get the PID of the sshd process and move it info the sshd tasks file:

cgroup]# echo $PID >> sshd/tasks

Ta-da, you're done. You can now rest assured that sshd will always have 50% of the CPU and 1G of memory.

score 1 · Answer 3 · answered Jun 01 '14 at 19:28

1

A more general solution to the problem of application resource usage is to run your applications in a container using Docker. You can then run containers with CPU and memory usage limits similar to cgroups.

docker run -c=10 -m=1g my-container

answered Jun 01 '14 at 19:28

Sean Fujiwara

111

score 1 · Answer 4 · edited Jun 13 '11 at 14:07

1

Use renice to get a higher priority for sshd, or check accounting. (acct) -> with this you can set resources for users, so run sshd with s

edited Jun 13 '11 at 14:07

Michael Mrozek

93,103
40
240
233

answered Jun 08 '11 at 06:01

MSpike

60

I would add ionice to that too, and probably start sshd with nice not renicing it later. – xenoterracide Jun 08 '11 at 13:03

How to limit resource usage to save CPU+RAM for a certain process?

4 Answers4

Linked