Why doesn't Linux automatically clear caches?

Question

Background

My question is basically a follow-up to this question and answer, and in particular, this comment.

Whenever I have to copy or rsync large amounts of files, the memory on my system tends to fill up, so I run a script like this (as root):

while true ; do echo "syncing" && sync && echo 3 > /proc/sys/vm/drop_caches && echo "done" && sleep 60 ; done ;

Questions

Could this harm the system in any way, or cause any negative effects? (In particular, what are the "risks" mentioned in the referenced comment?)
If the answer is "no" (which is what I suspect), then why does not Linux run this command automatically by default? (I don't see any noticeable change due to this, I only notice that I don't run out of memory...)

Yeah, I read that page, still what happens on my machine is that: 1. system gets slow 2. Linux dumps caches 3. system is fast again for some time. Whereas with my script it continues to run smoothly — Attilio, Mar 10 '20 at 16:06
I suspect there is something significant on your machine affecting memory usage. Linux does automatically free memory used for cache and buffering. It does it on a page by page basis instead of clearing it all at once. — virullius, Mar 10 '20 at 17:25
https://discuss.aerospike.com/t/how-to-tune-the-linux-kernel-for-memory-performance/4195 — ron, Mar 10 '20 at 18:25
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-tunables — ron, Mar 10 '20 at 18:26
It is advised to either reduce swappiness to 0 or not use swap. For low-latency operations, using swap to any extent will drastically slow down performance. — ron, Mar 10 '20 at 18:28

ctrl-alt-delor · Answer 1 · 2022-02-10T20:50:48.703

3

Because there is never any benefit in dropping caches. Forgetting stuff is not where the gain is. The gain is in reusing memory, this will happen anyway. As some point data will be evicted from memory.

However, doing these large jobs, may cause harm to performance: if data is cached, but never re-used, then it is causing other data to be evicted, with no benefit.

Therefore, you need to run the job with no caching (unfortunately I don't recall the command to do this).

edited Feb 10 '22 at 20:50

answered Mar 10 '20 at 18:16

ctrl-alt-delor

27,993

That is not true. As a small example, about 3 minutes ago I just ran into a scenario where sudo apt-get update would NOT work because it was reading the /etc/apt/sources file that was hold in cache (old version) I had to manually run command to clear cache in order for Apt to work with the NEW file. So dumb and excessively labor-intensive, as many things on the Linux environment at the moment. – Winampah Feb 09 '22 at 16:48
@Winampah If you can document this. It would be interesting. If it is true, then you have found a bug. Bugs do exist, but research has shown that UNIX has less bugs than MS-Windows, and Gnu/Linux has less bugs than UNIX. I suspect that you made a mistake, and then cleared the cache, then did it correct. – ctrl-alt-delor Feb 10 '22 at 20:55
Curiously, that happened while I was temporarily using a Xubuntu 20.10 ISO Live Session. But I haven't noticed it happening on the custom MX ISO's that I create (I only use Linux in Live Sessions, I don't ever install on disk) – Winampah Feb 14 '22 at 01:44
@Winampah I will put it down as mis-attribution. – ctrl-alt-delor Feb 15 '22 at 08:07

score 2 · Answer 2 · answered Mar 10 '20 at 18:58

Why doesn't Linux automatically clear caches?

It will.

ron> free -g
             total       used       free     shared    buffers     cached
Mem:           504        415         88          1          0        352
-/+ buffers/cache:         62        441
Swap:            0          0          0

Cached memory should still be considered free memory.

When free runs down to 0, then it will pull from cached.

Could this harm the system in any way, or cause any negative effects?

harm = no. Research echo 3 > /proc/sys/vm/drop_caches and you will see it is a non-destructive operation.

the negative effect would be the performance penalty incurred by having to read something off of disk that would have been in cache. For example write a C program that reads in a 10GB data file. The first time run it will be slow because it's reading from disk but anytime after will be much quicker because that data file is cached in ram. Drop the cache and next subsequent run of program reading data file will be slow like the first time it was run. This is easily observable and repeatable.

If the answer is no then why does not Linux run this command automatically by default?

You would have to elaborate how and where you see this happen. I am not aware of it on current linux such as RHEL/CentOS 7.7. But older linux's (such as pre 3.x kernel) it may have been a decision made by that distribution back when caching was not as robust as it is now (in my opinion). Using SLES 11.4 a few years ago I am pretty sure SuSE people did not write code to make that drop cache automatically happen, but for some work servers I had the folks we bought them from and configured SLES for us at the time has done a crontab to do periodic drop caching, so kind of a gray area as to why it's done. I suspect it's a housekeeping=good mentality and because it's never a destructive command can't really hurt. The few times I manually did echo 3 > drop_caches back on sles 11.4 was for troubleshooting as it never really solved anything where ultimately a reboot was needed.

Look into RHEL performance tuning guide, and tuning virtual memory. I don't know how much of this is RHEL specific versus being low level enough to be all linux kernel related and not specific to a linux distribution. Also what has changed from kernel 2.6 to 3.x to 4.x which I am sure is significant in this regard, so consider reading in to linux kernel release notes.

Why doesn't Linux automatically clear caches?

Background

Questions

2 Answers2