2

I'm trying to figure out why memory consumption started to constantly increase on my server during the last hours. I've tried to find the cause on the application level, but no success. That is why now looking into possible server cause. I'm not a pro in servers administration, so any help is appreciated. First common memory was eaten, now swap consumption is also constantly increasing.

My server runs on CentOS 7 with the kernel 3.10.0-514.26.2.el7.x86_64

SOLUTION
Finally, the issue was identified to be caused by a recently updated server library. The accepted answer is a good reminder, in the situation when you're stressed out by the memory usage, to trace back what had been changed in your system before the issue appeared.

Some tips I've been looking for and found to be very useful are described in How to find which processes are taking all the memory?

I'm listing below the commands that I used and may help in such situation.

ps auwx --sort rss - processes sorted by memory usage

ps -fu username - processes by a user

htop usage/analysis showed many hung application cron-launched processes in my case. I configured htop to output both PID and PPID, because I needed to correlate PPID to /var/log/cron logged processes.

free -m

    total        used        free      shared  buff/cache   available
Mem: 7565        6525         440          47         599         657
Swap: 8191        2612        5579

cat /proc/meminfo

MemTotal:        7747260 kB
MemFree:          253960 kB
MemAvailable:     498904 kB
Buffers:            6160 kB
Cached:           189076 kB
SwapCached:       467788 kB
Active:          5572588 kB
Inactive:        1258540 kB
Active(anon):    5498664 kB
Inactive(anon):  1185908 kB
Active(file):      73924 kB
Inactive(file):    72632 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       8388604 kB
SwapFree:        5686452 kB
Dirty:               104 kB
Writeback:             0 kB
AnonPages:       6168400 kB
Mapped:            68668 kB
Shmem:             48676 kB
Slab:             456672 kB
SReclaimable:     389064 kB
SUnreclaim:        67608 kB
KernelStack:        7232 kB
PageTables:       106848 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12262232 kB
Committed_AS:   10244216 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       26276 kB
VmallocChunk:   34359705340 kB
HardwareCorrupted:     0 kB
AnonHugePages:   5191680 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       83968 kB
DirectMap2M:     8300544 kB

How can I proceed to find if there's any issue in how the server is functioning or configured in terms of memory usage?

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
Y. E.
  • 123
  • The normal advice is to see how much memory is being used by processes, using either 'ps' or top. However it could be that you have tmpfs enabled, e.g. for /tmp and /run, and there are files in there which are using the space. – icarus Jul 18 '17 at 22:47
  • Thanks, icarus. Yes, I'm using ps and htop. I've found out there're hung application processes launched by cron, so going to investigate from here these ones. I've checked /tmp and /run - these are several Kb folders on my server - how could they influence memory? – Y. E. Jul 18 '17 at 23:05

1 Answers1

0

In general, you need to collect information over time to see what is leaking memory.
The easy target questions are:

1) What recent changes were made to the system?

2) What was updated?

3) Is there any information tying these changes to memory leaks.

After that you can use the programs available on your system. I don't have a Cent system but there must be a system monitor app that will show you how much memory the running processes are using.
The top can be used for the same thing on the command line. In both cases you need to monitor them and recognize what is growing.

If it is a server and has something like Nagios monitoring it, you can set up a memory monitor for different system apps and have it map the apps memory usage.

This is all very general stuff but it does give you a couple of directions to go.

Mark

Mark
  • 28
  • Mark, thank you. Yep, that what I'm doing - investigating and monitoring. The most difficult part there were no recent changes, at least if I'm not badly mistaken. I use top and htop all this time and finally started to get some decent results from this usage - I've updated my question with this information. I've found hung processes, that are cron-launched application ones. So going to investigate this direction. – Y. E. Jul 18 '17 at 23:00
  • If it continues to elude you you might try setting up cati to monitor apps of interest. Cati is what Nagios uses under the hood and can be quite handy for things like this, as long as you have an endless supply of aspirin. – Mark Jul 18 '17 at 23:10