How to regain control of my computer?

Question

I am running a database server on my computer. Sometimes it takes up so much memory resources that the system stop responding to keyboard and mouse inputs. Although I can move the mouse pointer and turn on and off Caps Lock, I can't do anything else beyond that.

Surprisingly, I am able to SSH to the computer, run the top command, kill a few processes and shutdown the database server to reclaim memory space. But even having done those things, the display remains in a semi-frozen state (mouse still animated).

Having reclaimed most of my memory, is it possible to regain usage of the system without having to reboot?

Answer 1

Freeing up resources should generally return the system to a normal functioning state, so it sounds to me like the system is still struggling to free up resources or, hasn't fully followed through on killing these processes. I'd investigate it further to find out if that's in fact the case. You can see, for example, if something is still writing data to the HDD. There are several tools that can assist with this, I'd start with fatrace to see if you can identify a process that's trying to finish up writing data to the disk.

Example

$ sudo fatrace | head -10
chrome(29486): W /home/saml/.config/google-chrome/Default/Extension State/017912.log
chrome(29486): CW /home/saml/.config/google-chrome/Default/File System/000/p/.usage
chrome(29486): W /home/saml/.config/google-chrome/Default/Extension State/017912.log
chrome(29486): W /home/saml/.config/google-chrome/Default/Extension State/017912.log
chrome(29486): W /home/saml/.config/google-chrome/Default/History-journal
chrome(29486): W /home/saml/.config/google-chrome/Default/History
chrome(29486): W /home/saml/.config/google-chrome/Default/History
chrome(29486): W /home/saml/.config/google-chrome/Default/History
chrome(29486): W /home/saml/.config/google-chrome/Default/History-journal
chrome(29486): W /home/saml/.config/google-chrome/Default/Extension State/017912.log

You'll want to run that command without the | head -10, that's just to show you the example here.

So what's wrong?

If you've ever looked at the output of free you've likely noticed the columns buffers and cache.

$ free 
             total       used       free     shared    buffers     cached
Mem:       7969084    6673652    1295432          0     118588     893916
-/+ buffers/cache:    5661148    2307936
Swap:      8011772    3104804    4906968

These are files that the system is aggressively loading into memory to maximize performance by using as much RAM as it can for this task. When the DB process (or which ever ones is consuming RAM) these files were pushed to swap (I'm assuming) and now cannot come back in since this other task is occupying the HDD's I/O.

What can be done to mitigate this?

One trick is to adjust the VM dirty ratio & VM dirty background ratio, which forces the system to start writing dirty pages of memory out to disk. This activity is often times what's causing a system to seemingly hang, especially in the UI. There are other reasons but this is one of the more frequented ones.

excerpt

By default the VM dirty ratio is set to 20% and the background dirty ratio is set to 10%. This means that when 10% of memory is filled with dirty pages (cached data which has to be flushed to disk), the kernel will start writing out the data to disk into the background, without interrupting processes. If the amount of dirty pages raises up to 20%, processes will be forced to write out data to disk and cannot continue other work before they have done so.

Here's how you can check on your system's current settings:

$ sudo sysctl -a | grep 'dirty.*ratio'
vm.dirty_background_ratio = 10
vm.dirty_ratio = 20

To override these settings you can create the following file, /etc/sysctl.d/dirty_ratio.conf with the following content:

vm.dirty_ratio = 10
vm.dirty_background_ratio = 5

This will cause your system to be more aggressive about writing changes out as the occur. You can activate these changes immediately like so:

$ sudo sysctl -p /etc/sysctl.d/dirty_ratio.conf

Will this resolve the issue?

In my experience you can tweak these values but the true issue is your system is likely just not up to the task(s) you're asking it to perform.

References

• LINUX PERFORMANCE IMPROVEMENTS