Linux Cache cannot be cleared completely, it fills the entire RAM after some days

Question

I use a laptop for work and therefore I only do a suspend to RAM rather than shutdown everyday. so I do not have to start each application again every day.

after some days of "run time" I see that Cache gets filled up, but I'm unable to clear it completely. this leads to swap usage and later to OOM.

# echo 3 > /proc/sys/vm/drop_caches; free -wht
              total        used        free      shared     buffers       cache   available
Mem:           23Gi       2.2Gi        15Gi       4.7Gi       0.0Ki       5.2Gi        15Gi
Swap:         2.0Gi          0B       2.0Gi
Total:         25Gi       2.2Gi        17Gi
# free -wht
              total        used        free      shared     buffers       cache   available
Mem:           23Gi       4.4Gi       457Mi       9.1Gi       3.5Gi        14Gi       9.2Gi
Swap:         2.0Gi       0.0Ki       2.0Gi
Total:         25Gi       4.4Gi       2.4Gi
# echo 3 > /proc/sys/vm/drop_caches; free -wht
              total        used        free      shared     buffers       cache   available
Mem:           23Gi       4.3Gi       9.0Gi       9.0Gi       0.0Ki       9.8Gi       9.4Gi
Swap:         2.0Gi       0.0Ki       2.0Gi
Total:         25Gi       4.3Gi        11Gi
# echo 3 > /proc/sys/vm/drop_caches; free -wht
              total        used        free      shared     buffers       cache   available
Mem:           23Gi       4.3Gi       9.0Gi       9.0Gi       0.0Ki       9.8Gi       9.4Gi
Swap:         2.0Gi       0.0Ki       2.0Gi
Total:         25Gi       4.3Gi        11Gi
# uptime
 10:57:09  up 3 days 19:26,  5 users,  load average: 0.89, 1.94, 1.69
# uname -a
Linux xxxxx-00164 5.3.18-150300.59.76-preempt #1 SMP PREEMPT Thu Jun 16 04:23:47 UTC 2022 (2cc2ade) x86_64 x86_64 x86_64 GNU/Linux
# 

I could not yet figure out how to check what application or files are stuck in the cache. I assumed that 3 > drop_caches removes all entries of the file cache. but as my laptop runs some hours more, the drop cache action will free less every time. the output above spans accross ~8h hours. what remains in Cache is missing from Available, although I tried to clear the Cache twice within 1s (the last two echo commands)

how to analyse such a behavior? what remains in Cache??

Many Thanks! Stefan K.

Answer 1

after some days of "run time" I see that Cache gets filled up, but I'm unable to clear it completely. this leads to swap usage and later to OOM.

I think you're chasing the wrong problem. The drop_caches command can't drop anything that's in use — that's why it doesn't go to zero. But it's normally not a problem. You can probably find a dozen answers on this site (like How do you empty the buffers and cache on a Linux system?) telling you that this normally pointless.

But that said, I don't think any of these explain why echo 3 > /proc/sys/vm/drop_caches isn't making the cached number go right to zero. I know of at least two things that can have impact: files in tmpfs (including ones which are still open but may be deleted (see this answer on Serverfault) and zram (see zramctl command). There may be some others.

The basic problem isn't the cache — something is using your memory for real.