You looked at a list of processes and their memory usage. But there was a problem. You were not looking at the complete list.
gnome-system-monitor
only shows "My Processes" by default. To see processes owned by all system users, including the root
user, click the menu icon in the top right (three dots in a vertical line). Change the selection from "My Processes" to "All Processes".
- Used v.s. available system RAM, as a single number
- Analyzing /proc/meminfo
- Other tools you could use - install
atop
today 8-)
1. Used v.s. available system RAM, as a single number
I see your ruby code subtracts MemAvailable
from MemTotal
. This is the exact same calculation used by gnome-system-monitor
, when it shows the system is using "1.5 GiB (41.4%) of 3.7 GiB".
You are right to use either gnome-system-monitor
or your manual calculation, at least as a first approximation. The MemAvailable
number includes basically all the reclaimable "cache". I.e. MemAvailable
includes a type of "cache" that is "available" to be reclaimed, as soon as a program asks for more memory than you have free.
Side note: There is another type or meaning of "cache", which is not reclaimable. When you look at the Cache
/ "cache" number, it is usually reported as including Shmem
/ "shared". The Shmem
part is not reclaimable cache. The confusion happens because Shmem
was cleverly implemented using the kernel "page cache".
Another way to quickly check "available" is free -h
.
The free
command also shows "shared", swap usage etc. Your system documentation should list the output fields and available options, i.e. in man free
. Some of the other fields can be misleading:
- The "used" field in the
free
command (currently) does not include "shared". This can be very confusing. Ignore the "used" field.
- The "cached" value shown by
free
suffers from the problem noted above.
- If
free
does not show "available", your system is ancient. Consult your ancient documentation.
2. Analyzing /proc/meminfo
Thank you for the full output of cat /proc/meminfo
. This often helps to find specific answers (or any answer at all). If you wanted to look through how MemAvailable
is calculated yourself, you could read the first section of my answer here: Is "Cached" memory de-facto free?
AnonPages
In your example meminfo
, you have AnonPages: 924844 kB
(0.9 GB). AnonPages
is one of the terms that reduces MemAvailable
.
When AnonPages
is increasing, that should indicate an increase in the "RES", or "RSS" ("Resident" in ram "Set Size") of some number of running programs. But RSS can be misleading, because some memory is shared:
You cannot add up RSS, because that would double-count shared memory. You have to add up PSS, the proportional RSS after accounting for sharing. The smem
command can show PSS, and also calculate totals. For example:
sudo smem -t > p; head -n1 p; echo; tail -n17 p
- Look at per-process memory usage. The tail
part will show the top 15 processes, followed by a line with total PSS etc.
smem -t -U ^sourcejedi$ > U; head -n1 U; echo; tail -n17 U
- Look at memory usage of processes belonging to my user, "sourcejedi".
sudo smem -t -u
- Look at memory grouped by user. This might be useful to distinguish your login session(s), from some system daemons that run as their own user. E.g. packagekitd runs as the root
user, and can use hundreds of megabytes.
smem -t -P firefox
- Look at the memory usage of my web browser :-).
sudo smem -t -m > m; head -n5 m; echo; tail m
- Look at memory grouped by the mapping name - the name of the cached file, or "<anonymous>" or "[heap]".
Process "resident" memory includes both "anonymous" memory, and some cached files. smem -m
cannot show all cached files, only certain types of files that are currently in use. Specifically, files which a program has mapped into virtual memory. This includes program code, library code, and files mapped using mmap().
Shmem
You also have Shmem: 374848 kB
(0.4 GB). I mentioned Shmem
/ "shared" above. This is another term that reduces the "available" memory. (It is not reclaimable cache). This is fairly normal, but you could try to see what it is.
Some shared memory is visible as the memory of individual processes. If the shared memory is mapped by a process, then it should be counted in the RSS / PSS. See above. The "mapping name" can sometimes be useful here (e.g. smem -t -m
).
Shmem
includes files on mounted tmpfs
. You can check all mounted tmpfs
using df -t tmpfs
.
Depending on your system, Shmem
could include some graphics buffers. I found a way to check their current size on my system (Intel graphics): Can I see the amount of memory which is allocated as GEM buffers? I would love to know if you find a different way to check on your system!
I read that Shmem
memory leaks in some other graphics driver can be associated with very large VIRT (aka VSIZE) for Xorg. Linux using whole swap, becoming unresponsive while there is plenty of free RAM
Other memory usage?
MemTotal - MemAvailable = 1796088 kB
(1.8 GB)
AnonPages: 924844 kB
(0.9 GB)
Shmem: 374848 kB
(0.4 GB)
Of the remaining 0.5 GB, I can see a few more small uses under 0.1 GB overall. The kernel also reserves a few percentage points of margin for itself (see "low water mark"), but I guess it would be 0.2 GB or less on your system. So there is a bit more usage that I am not sure about.
Kernel slab memory
"Unreclaimable slab" memory is another terms that reduces MemAvailable
. You do not have very much: SUnreclaim: 55044 kB
(0.05 GB).
You can also run slabtop to see the list of slabs. AFAICT, slabtop doesn't give stats on slabs as reclaimable or unreclaimable. But I can usually guess, and if there is some suspicious slab I guess you can look it up by name.
3. Other tools you could use - install atop
today 8-)
smem
might be overkill. Sometimes, all you need is top
, or your favourite alternative, and to know how to sort by resident memory. (Although gnome-system-monitor
might not be a good choice for this. I think it does not really show enough).
Sometimes when you have a performance problem, you need to look at disk reads & writes instead. You could use sudo iotop
.
Sometimes, you would like to have a log of per-process memory usage, so you can look at why you ran out of memory and slowed the system to a crawl...
atop
is a nifty little tool that can do all the above. If this sounds useful, I suggest you install it straight away. Then you can learn about it when you need it :-).
sudo atop -R
shows "PSIZE" (same meaning as "PSS"). The atop
package includes a background service that runs atop with a ten minute interval. You can use atop -r ...
to open the log files, which are stored under /var/log/atop/.
require 'objspace' ; puts ObjectSpace.memsize_of(Array.new(200_000_000, &:itself))
and I saw that when the RAM usage hits 2.5 GiB, it suddenly drops to 2.3 GiB and then I killed the program, saw that 1.2 MiB swap is written, and the RAM usage dropped to 1.2 GiB. – 15 Volts Jun 14 '19 at 08:20