8

My Mint 18.1 system often freezes for approximately 10 minutes due to running out of memory, as per a widget which shows 100% memory use at freeze time. SysRq + "Raising Skinny Elephants Is Utterly Boring" combinations do not work.

I have tried setting swap priority to -1, which does not solve the freezes.

My swap partition is barely used, even on high loads and with vm.swappiness=100, even when 95%+ of RAM is in use. Would forcing an increase of swap use solve the freezing, and if so, how could I force more use of swap?

->free -m
              total        used        free      shared  buff/cache   available
Mem:          15874       12243         412        1255        3218        1724
Swap:         16207           5       16201

->swapon -s
Filename                Type        Size    Used    Priority
/dev/dm-3             partition 16596476    6068    -1

enter image description here

My system:

NAME="Linux Mint"
VERSION="18.1 (Sonya)"
K7AAY
  • 3,816
  • 4
  • 23
  • 39
OlehZiniak
  • 163
  • 1
  • 7
  • 3
    Could you show details of the freezing you’re seeing, and why it’s caused by running out of RAM? Regarding vm.swappiness, it doesn’t cause the system to swap “pre-emptively”, it causes the system to swap instead of reclaiming cache when it needs more memory; depending on the scenario, increasing it might well contribute to the stalls you’re seeing. – Stephen Kitt Mar 19 '19 at 10:40
  • 1
    @OlehZiniak, as the manpage of swapon suggests, priority should be something between 0 and 32767. Try to edit your fstab and put a 0 priority on your swap partition or, swapoff and swapon /dev/dm-3 -p 0. I know it sounds weird but, it isn't the first time i hear that negative values on swap priority somehow makes a system not to use swap at all. Try that and if it works ill post my comment as an answer... –  Mar 19 '19 at 12:02
  • 3
    @nwildner what version of the manpage are you looking at? Mine, which matches this, says that the priority can be between -1 and 32767, and that -1 is the default priority. (And that’s incorrect, priorities can have lower values still; see the syscall for the allocation policy. The default setup, starting at -1 with low priority allocation, means that the second swap area has priority -2, etc.) – Stephen Kitt Mar 19 '19 at 12:21
  • 1
    I'm looking directly at the manpages of a CentOS 6.11, where the "lowest possible priority" was 0, but now it seems to be -1(set by kernel). Just guessing here since i use priority 0 at home and at work(Arch Linux x CentOS) explicitly set at fstab and never had that "not swapping" behavior. If that shows true, it could be a bug on this priority mechanism. –  Mar 19 '19 at 12:54
  • 4
    Not helping the question, but the swap priority matters only when there is more than one swap space in use, because it's about relative priority between those spaces (and using the same positive value allows swap load balancing) – A.B Mar 19 '19 at 18:20

3 Answers3

3

You have got two problems here, apparently.

The REISUB sequence not working could be caused by magic SysRq not being activated - check out

cat /proc/sys/kernel/sysrq

# List of possible values in /proc/sys/kernel/sysrq:

#  0 - disable sysrq completely
#  1 - enable all functions of sysrq
#    >1 - bitmask of allowed sysrq functions (see below for detailed function description):
#      2 - enable control of console logging level
#      4 - enable control of keyboard (SAK, unraw)
#      8 - enable debugging dumps of processes etc.
#     16 - enable sync command
#     32 - enable remount read-only
#     64 - enable signalling of processes (term, kill, oom-kill)
#    128 - allow reboot/poweroff
#    256 - allow nicing of all RT tasks

You want 4 (R) + 64 (E,I) + 16 (S) + 32 (U) + 128 (B) = 244 for classic REISUB (the effect is the same as the skinny elephant version; as a mnemonic, it's BUSIER spelled backwards).

For the swappiness of the system: it's working as expected. The cause of the behaviour is that you have lots of memory used - that could be the 95% you're getting - and then something else is requesting a whole lot of new RAM, forcing most of that 95% onto swap and fighting all the way.

The best thing you could do is install more RAM. Otherwise, see whether you can tame either the small processes building up to that 95%, or the big one eating a further 40-50%. The top utility might help you there.

As a fancy solution you could increase apparent swappiness and keep the small fry "pared down" with some process that monitored the system and, if the big hog was not running and the memory footprint was too large, started allocating and releasing large swaths of memory - say, all remaining memory plus 64M, to force 64M of small fry onto the cache, then +128M, and so on, until the allocation delay grew beyond a given time threshold or the big hog started.

This answer could give you a good start if you really want to do that.

LSerni
  • 4,560
3

Just in short, swap won't help you, the most effective solution to this problem is expanding the memory if you keep your system environment unchanged.

Now answer the questions you just mentioned.

1. Why memory doesn't swapout despite 'vm.swappiness=100'?

Firstly, you should know how swap works in Linux. When the kernel feels memory insufficient, there are two ways to reclaim memory to relief the situation.

  1. swapout anonymous pages
  2. reclaim cached pages

and 'vm.swappiness' just control how aggressive the kernel will swap memory pages (swapout more anonymous pages or reclaim more cached pages)

So, here comes another question, how does the kernel determine which anonymous page to be swapped? The strategy is complicated, you can refer the codes here according to your kernel version. Summarize the strategy in a single sentence, swapout the most inactive pages. What if there is no inactive page that the kernel thinks could be swapped?

That's the reason why your system doesn't swapout as much memory as you thought.

2. Does forcing memory swap help?

No. And there is no common way to force swap memory, the kernel takes over it.

Shellmode
  • 101
0

If the system is a laptop, its keyboard sysreq key might not work unless an additional function key, (which acts like a shift key), is pressed.

Whenever an app (or set of apps) is hogging more than 16GB RAM, it's usually best to find a less bloated app.

agc
  • 7,223