5

I'm running a desktop system that quite regularly suffers from lack of memory, this prompted me to investigate what causes the issue in the first place.

Problem is, there's no single process that eats the memory, yet the system doesn't show it as available. What's more, the system does swap so it looks like the memory pressure is real. What's puzzling, is that the usage goes to normal (~1GB used) after I log out and back again so it looks like some weird interaction between userland and kernel and not a memory leak.

In short:

  • memory reported as used by free, excluding cache/buffers: 3173960 kB
  • sum of USS of all applications: 2413952 kB
  • SLAB size: 158968 kB
  • zram (after compression): 75992 kB

That gives, 3173960-2413952-158968-75992 = 525048 kB unaccounted memory usage.

What I'm missing or not counting?

Sum of applications memory usage:

# smem -t | sed -n '1p;$p'
  PID User     Command                         Swap      USS      PSS      RSS 
  108 6                                      244524  2413952  2461340  2648488

Memory usage as reported by free:

# free -k
             total       used       free     shared    buffers     cached
Mem:       4051956    3449748     602208          0      26548     249240
-/+ buffers/cache:    3173960     877996
Swap:      4051952     242592    3809360

General memory statistic:

# cat /proc/meminfo 
MemTotal:        4051956 kB
MemFree:          612260 kB
Buffers:           26636 kB
Cached:           249304 kB
SwapCached:       107892 kB
Active:          1774004 kB
Inactive:         885268 kB
Active(anon):    1712484 kB
Inactive(anon):   710788 kB
Active(file):      61520 kB
Inactive(file):   174480 kB
Unevictable:        9332 kB
Mlocked:            9332 kB
SwapTotal:       4051952 kB
SwapFree:        3809368 kB
Dirty:                40 kB
Writeback:             0 kB
AnonPages:       2343292 kB
Mapped:            95288 kB
Shmem:             36396 kB
Slab:             158968 kB
SReclaimable:      53900 kB
SUnreclaim:       105068 kB
KernelStack:        3528 kB
PageTables:        43600 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6077928 kB
Committed_AS:    4013288 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      139852 kB
VmallocChunk:   34359570976 kB
HardwareCorrupted:     0 kB
AnonHugePages:    641024 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     2310848 kB
DirectMap2M:     1882112 kB

Swaps are on zram:

# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       2025976 121252  100
/dev/zram1                              partition       2025976 121324  100

# awk ' { print $0 / 1024; sum+=$0 } END { print "sum:" sum/1024 } ' /sys/block/zram*/compr_data_size
37962.4
38030.1
sum:75992.5
Hubert Kario
  • 560
  • What is "periodic lack of memory"? Does it crash, does the Out of Memory killer (OOM) kick in, ...? Any relevant log entries? [If none of the above, your system is working fine, and you don't have to worry. The sysstat package provides the sar command, you can set it up to record (and analyze) many performance variables in minute detail.] – vonbrand Jan 21 '13 at 04:31
  • OOM kicks in. And I think when it starts, the situation is much worse (as in 1G of unaccounted for memory). It is working fine (when it is not actively swapping, because then I can't move the mouse, let alone use applications), I just want to know where my memory is getting used up. – Hubert Kario Jan 21 '13 at 12:32
  • 1
    What gets killed? The OOM heuristics are rather good at blaming the culprit (but some processes, like X, your display manager and some system processes, get a free pass). If it is relatively common, you could try to write out the memory usage for processes periodically and check if something is balooning up. Anything long-running that is self-built or patched? BTW, log out and log in again normalizes stuff definitely points at some userland gobbling up memory. A random thought: Any of those misguided "index all user files" background processes running? – vonbrand Jan 21 '13 at 12:44
  • Well, the OOM not really kicks in, it's just that sometimes it starts swapping so hard that the only way to recover is to reset the system as nothing else works (including remote ssh login). I'm using KDE and Nepomuk is the first thing I removed. – Hubert Kario Jan 21 '13 at 17:50
  • This could also be a bug see this other question: http://unix.stackexchange.com/questions/39721/ram-usage-doesnt-add-up-freeused-total If you use the meminfo output from this question and feed it to smem, you get roughly 86 MB of kernel cache memory and 1 GB of kernel noncache memory. We need to know what services and their versions you are using. Also a "screenshot" of top and slabtop would be necessary. – Huygens Mar 04 '13 at 07:51
  • yes, it is similar, but I don't use NFS at all and moreover, I had this problem while running 3.7 and 3.6 – Hubert Kario Mar 05 '13 at 17:25
  • @HubertKario Are you using tmpfs or ramfs? – Huygens Mar 05 '13 at 22:37
  • @Huygens: yes, I was using it, I don't know if its size grew – Hubert Kario Mar 08 '13 at 12:42
  • 1
    @HubertKario then that is something to investigate :) – Huygens Mar 08 '13 at 22:16

3 Answers3

1

The problem

4 GB of RAM (physical memory) and that you have 2 zram device of maximum 2,025,976 kB (roughly 2 GB each). zram is using the available memory, I do not know exactly the internal but whatever the mechanism I can clearly imagine a scenario where Linux page out (= put some memory from the RAM to zram) to get some more free space but then the zram usage in memory is growing, so it would further page out, which would result in further increase of zram usage, and so on until zram is consuming all your physical memory.

I guess there is a threshold on any system under which the paging out won't stress the kernel to the point I describe above, so that zram improve performance.

Insights

When your system wants to swap 100 MB, what happens is that it puts this 100 MB in zram. Let's say it gets compressed to 50% less, so 50 MB. It means that your system wanted to free 100 MB but only 50 MB got freed. Now Linux is clever in that when it has paged out chunk of memory (so put them in the swap) but need them again, it can do some "optimisation", it can page in again this memory but keep it in the swap as well, so if quickly after it would need to page out these part of the memory it could avoid an expenive write to the swap file. So in your case, it could be that Linux keeps the 100 MB in zram and put them back in normal RAM, so the system consumes 150 MB for awhile. If this is repeated for bigger program with less compressible data, this could quickly become a nightmare, imagine a 300 MB chunk of RAM that would be paged out, and use 120 MB in each zram swap. It means that Linux wanted to free 300 MB of the RAM for other purpose, but has only freed (300-120-120=60) 60 MB, it might then try to page out further pages, and so on, with the problem that you have 2 zram that can use up to 2 GB of RAM each, thus eating all your memory.

Conclusion and solution

So is zram crap? No, not at all, the problem is that you configured zram to have a total size of exactly your physical RAM and that's the problem. You should not configure zram to use more than 25% IMHO of your physical RAM, which means you would have to rely still in a hard disk swap solution once zram swap is filled up.

A simple solution would be to reduce both zram to handle each 500 MB max and add a swap file of roughly 2-3 GB, to allow the kernel to free really unused pages from zram to this swap file. The swap file won't use the RAM and dimish the pressure on it.

Some information on how to set your zram disk size.

Huygens
  • 9,345
  • First: while your in-depth explanation of zram is not far from reality, it's supposed to be used without real swap. Second: disabling zram didn't really help. The problem is probably caused by some bug in radeon driver, unfortunately I won't have time to debug it in near future. – Hubert Kario Mar 02 '13 at 19:55
  • I do have an old PC which I am still using regularly with only 1 GB of RAM, an old Sempron 2800+ and an AGP Radeon card. Memory is extremely limited nowadays, by simply browsing the internet you quickly start using the swap. On this machine, I have set up zram, but I also kept my swap partition. I could not give too much of the 1 GB to zram without limiting my physical RAM, so I gave only 256 MB, Basically it was like increasing the RAM from 1 GB to something a bit higher (with a little overhead ;-) ) and that was just enough for my usage, but I still keep on swaping to disk! – Huygens Mar 03 '13 at 08:15
  • @HubertKario Neither the kernel documentation on zram nor the compcache (the original project) are telling/advising to any hard disk swap. I guess this would depend on the workload and resource need that it implies. On Linux you can give priority for each swap, like I said I have one machine which is using zram, it has the highest priority for zram swap, and the lowest for hard disk swap, and I can clearly see one filling up before the other is even touched. An example of how to configure it: https://wiki.archlinux.org/index.php/Swap#Priority – Huygens Mar 03 '13 at 08:24
  • @HubertKario as you know I have set up a test VM for zram. I have found out that under memory stress, the kernel will start swaping, and instead of having a linear filling up of the swap it was much faster/easy to do so! After removing some of the memory pressure I still had swap > free memory, so the kernel had to keep some swap. I reactivated my hard disk swap and removed zram. This force the memory of the 4 zram disks to be put back in RAM and at the end I ended up with: 0 MB swap used and 1 GB free memory!! zram can be handy but also a memory hog! – Huygens Mar 03 '13 at 18:45
  • I appreciate your input, but as I've said, disabling zram and enabling regular swap didn't help, ergo zram isn't the problem. – Hubert Kario Mar 03 '13 at 21:06
  • Is your question just about the counting of memory? Or are you looking for which process is using the memory? I am not sure how to help you further. – Huygens Mar 03 '13 at 21:31
  • @Hyugens: There's 525048 kB of memory that I don't know how it's used, it's not SLAB, it's not cache, it's not buffers and it's not user applications. The question is, how can I get detailed info about kernel memory usage? – Hubert Kario Mar 05 '13 at 17:16
  • You know, it is the top-down or bottom-up approach. You want one where you start form the kernel detailed info and go up to the app, I would start from the apps and try identifying which is the culprit. Both approach are correct! But I can only help you when drilling from the apps to the kernel I'm afraid, as we have hit the limit of my kernel knowledge now... – Huygens Mar 05 '13 at 22:35
  • 1
    I started with the applications down: there wasn't any application that was using significantly more memory after week of uptime than when it was freshly started, that's why I started searching what's happening in the kernel – Hubert Kario Mar 08 '13 at 12:43
0

The free -tm command also shows the swap usage by files backed via tmpfs. If you can empty your /tmp folder, the memory usage of smem -t -k should be simular to free -tm.

vdboor
  • 2,348
0

I've found that smem does see the used memory, but only in "system" mode:

# smem -tw
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory       1200240     379444     820796 
userspace memory            2101184     136800    1964384 
free memory                  750532     750532          0 
----------------------------------------------------------
                            4051956    1266776    2785180 
# free -k
             total       used       free     shared    buffers     cached
Mem:       4051956    3298200     753756          0      31664     425552
-/+ buffers/cache:    2840984    1210972
Swap:      4051952     237368    3814584

which gives a (hardly any) difference of around 55MiB between non cache used memory between free and smem.

Hubert Kario
  • 560
  • smem see the memory in system mode because it accidentally compute the zram real usage in the kernel dynamic memory noncache. Your swap usage is displayed in the "swap" line by free but also indirectly in the above lines reported by free too! – Huygens Mar 03 '13 at 18:49
  • and that's probably the 55MiB difference I'm seeing between free and smem (as zram compresses the data, and I remember seeing a 4 fold size reduction), thing is, it's not the problem I'm trying to diagnose. The issue at hand is why my noncache kernel memory is big in the first place (even if we assume that zram memory is uncompressed, it's still over 600MiB that's being used, that is, over an eight of total memory!). – Hubert Kario Mar 03 '13 at 21:04
  • Your 55 MB difference could be simply explained because smem forgets to add the Shmem field to the user space used memory, you seem to had around 36 MB of those but as time of "snapshotting" the commands might differ it could explain the difference. – Huygens Mar 03 '13 at 21:16
  • What I wanted to say is that smem uses the content of meminfo to compute the kernel dynamic memory line values. However, meminfo does not expose all types of memory (try to add them to see if they would add up). Any weird memory usage (such as zram could produce, but other programs/modules could do also other stuff) end up counted in kernel noncache. Do not focus on the output of smem perhaps your memory consumption does not come form the kernel but a userspace apps. We need to know more the type of applications you are running to help you further. – Huygens Mar 03 '13 at 21:18
  • In addition to zram, are you using tmpfs? Or any other ramfs like features? – Huygens Mar 04 '13 at 08:45