Python consume too much SWAP

Question

Please take a look on output of command lsof -p PID where PID is python process.

COMMAND  PID USER   FD   TYPE             DEVICE  SIZE/OFF     NODE NAME
python  1459 root  cwd    DIR              253,0      4096      128 /
python  1459 root  rtd    DIR              253,0      4096      128 /
python  1459 root  txt    REG              253,0      7144     1623 /usr/bin/python2.7
python  1459 root  mem    REG              253,0     20088 17058062 /usr/lib64/sasl2/libplain.so.3.0.0
python  1459 root  mem    REG              253,0     20056 16847914 /usr/lib64/sasl2/liblogin.so.3.0.0
python  1459 root  mem    REG              253,0     28272 16847886 /usr/lib64/sasl2/libsasldb.so.3.0.0
python  1459 root  mem    REG              253,0     20064 16847883 /usr/lib64/sasl2/libanonymous.so.3.0.0
python  1459 root  mem    REG              253,0    611424   175002 /usr/lib64/python2.7/site-packages/_cproton.so
python  1459 root  mem    REG              253,0     31344  8453741 /usr/lib64/libnss_dns-2.17.so
python  1459 root  mem    REG              253,0     54552 25169653 /usr/lib64/python2.7/lib-dynload/pyexpat.so
python  1459 root  mem    REG              253,0    173320  8431306 /usr/lib64/libexpat.so.1.6.0
python  1459 root  mem    REG              253,0     47816 25320463 /usr/lib64/python2.7/lib-dynload/_elementtree.so
python  1459 root  mem    REG              253,0   1509584  8453762 /usr/lib64/libxml2.so.2.9.1
python  1459 root  mem    REG              253,0   1156600  8431593 /usr/lib64/libglib-2.0.so.0.5600.1
python  1459 root  mem    REG              253,0     50200   288064 /usr/lib64/python2.7/site-packages/_sqlitecache.so
python  1459 root  mem    REG              253,0     85200 25403939 /usr/lib64/python2.7/lib-dynload/_sqlite3.so
python  1459 root  mem    REG              253,0     25664 25169660 /usr/lib64/python2.7/lib-dynload/termios.so
python  1459 root  mem    REG              253,0     19384  8431778 /usr/lib64/libgpg-error.so.0.10.0
python  1459 root  mem    REG              253,0     70856  8445273 /usr/lib64/libassuan.so.0.4.0
python  1459 root  mem    REG              253,0    212120  8635820 /usr/lib64/libgpgme.so.11.8.1
python  1459 root  mem    REG              253,0     83568  8635824 /usr/lib64/python2.7/site-packages/gpgme/_gpgme.so
python  1459 root  mem    REG              253,0     67888   288069 /usr/lib64/python2.7/site-packages/lzma.so
python  1459 root  mem    REG              253,0     39848 25214104 /usr/lib64/python2.7/lib-dynload/bz2.so
python  1459 root  mem    REG              253,0     32520   288076 /usr/lib64/python2.7/site-packages/xattr.so
python  1459 root  mem    REG              253,0     11392  8388746 /usr/lib64/libfreebl3.so
python  1459 root  mem    REG              253,0     40600  8453729 /usr/lib64/libcrypt-2.17.so
python  1459 root  mem    REG              253,0    121208  8395594 /usr/lib64/libsasl2.so.3.0.0
python  1459 root  mem    REG              253,0    352584  8474165 /usr/lib64/libldap-2.4.so.2.10.7
python  1459 root  mem    REG              253,0     61952  8474163 /usr/lib64/liblber-2.4.so.2.10.7
python  1459 root  mem    REG              253,0    168336  8445735 /usr/lib64/libsmime3.so
python  1459 root  mem    REG              253,0    370584  8457203 /usr/lib64/libssl3.so
python  1459 root  mem    REG              253,0    186728  8389393 /usr/lib64/libssh2.so.1.0.1
python  1459 root  mem    REG              253,0    208928  8431938 /usr/lib64/libidn.so.11.6.11
python  1459 root  mem    REG              253,0    439320  8395543 /usr/lib64/libcurl.so.4.3.0
python  1459 root  mem    REG              253,0     70712     8849 /usr/lib64/python2.7/site-packages/pycurl.so
python  1459 root  mem    REG              253,0     13440 25320450 /usr/lib64/python2.7/lib-dynload/_bisectmodule.so
python  1459 root  mem    REG              253,0     12352 25175843 /usr/lib64/python2.7/lib-dynload/grpmodule.so
python  1459 root  mem    REG              253,0     19872  8593218 /usr/lib64/librpmsign.so.1.2.2
python  1459 root  mem    REG              253,0      7176 25384603 /usr/lib64/python2.7/site-packages/rpm/_rpms.so
python  1459 root  mem    REG              253,0    120056  8394251 /usr/lib64/libmagic.so.1.0.0
python  1459 root  mem    REG              253,0    146784  8593216 /usr/lib64/librpmbuild.so.3.2.2
python  1459 root  mem    REG              253,0     17032 25181303 /usr/lib64/python2.7/site-packages/rpm/_rpmb.so
python  1459 root  mem    REG              253,0    555888  8396265 /usr/lib64/libfreeblpriv3.so
python  1459 root  mem    REG              253,0    753280  8409648 /usr/lib64/libsqlite3.so.0.8.6
python  1459 root  mem    REG              253,0    278192  8428927 /usr/lib64/libsoftokn3.so
python  1459 root  mem    REG              253,0     43712  8453755 /usr/lib64/librt-2.17.so
python  1459 root  mem    REG              253,0     23968  8445689 /usr/lib64/libcap-ng.so.0.0.0
python  1459 root  mem    REG              253,0    251888  8396532 /usr/lib64/libnspr4.so
python  1459 root  mem    REG              253,0     15800  8409544 /usr/lib64/libplds4.so
python  1459 root  mem    REG              253,0     20096  8409540 /usr/lib64/libplc4.so
python  1459 root  mem    REG              253,0    198968  8389372 /usr/lib64/libnssutil3.so
python  1459 root  mem    REG              253,0     19896  8431628 /usr/lib64/libattr.so.1.1.0
python  1459 root  mem    REG              253,0    127112  8416992 /usr/lib64/libaudit.so.1.0.0
python  1459 root  mem    REG              253,0    193864  8431855 /usr/lib64/liblua-5.1.so
python  1459 root  mem    REG              253,0    157400  8431193 /usr/lib64/liblzma.so.5.2.2
python  1459 root  mem    REG              253,0     41456  8431389 /usr/lib64/libpopt.so.0.0.0
python  1459 root  mem    REG              253,0     99952  8442798 /usr/lib64/libelf-0.176.so
python  1459 root  mem    REG              253,0     68192  8445628 /usr/lib64/libbz2.so.1.0.6
python  1459 root  mem    REG              253,0   1257808  8442785 /usr/lib64/libnss3.so
python  1459 root  mem    REG              253,0    178928  8474328 /usr/lib64/librpmio.so.3.2.2
python  1459 root  mem    REG              253,0   1845784  8549532 /usr/lib64/libdb-5.3.so
python  1459 root  mem    REG              253,0     37064  8431624 /usr/lib64/libacl.so.1.1.0
python  1459 root  mem    REG              253,0     20048  8416910 /usr/lib64/libcap.so.2.22
python  1459 root  mem    REG              253,0    432504  8474326 /usr/lib64/librpm.so.3.2.2
python  1459 root  mem    REG              253,0    108664 25169350 /usr/lib64/python2.7/site-packages/rpm/_rpm.so
python  1459 root  mem    REG              253,0     21256 25169640 /usr/lib64/python2.7/lib-dynload/_localemodule.so
python  1459 root  mem    REG              253,0     24120 25169109 /usr/lib64/python2.7/lib-dynload/zlibmodule.so
python  1459 root  mem    REG              253,0     20736  8635780 /usr/lib64/python2.7/site-packages/rhsm/_certificate.so
python  1459 root  mem    REG              253,0     48152 25169645 /usr/lib64/python2.7/lib-dynload/arraymodule.so
python  1459 root  mem    REG              253,0     20064  8388750 /usr/lib64/libuuid.so.1.3.0
python  1459 root  mem    REG              253,0     29184 25169656 /usr/lib64/python2.7/lib-dynload/selectmodule.so
python  1459 root  mem    REG              253,0     86944 25169647 /usr/lib64/python2.7/lib-dynload/datetime.so
python  1459 root  mem    REG              253,0     19216 25175840 /usr/lib64/python2.7/lib-dynload/fcntlmodule.so
python  1459 root  mem    REG              253,0     16344 25391670 /usr/lib64/python2.7/lib-dynload/_randommodule.so
python  1459 root  mem    REG              253,0     22824 25169637 /usr/lib64/python2.7/lib-dynload/_hashlib.so
python  1459 root  mem    REG              253,0     37368 25173236 /usr/lib64/python2.7/lib-dynload/math.so
python  1459 root  mem    REG              253,0    156856 25169638 /usr/lib64/python2.7/lib-dynload/_io.so
python  1459 root  mem    REG              253,0     32392  8425851 /usr/lib64/libffi.so.6.0.1
python  1459 root  mem    REG              253,0    126952 25320458 /usr/lib64/python2.7/lib-dynload/_ctypes.so
python  1459 root  mem    REG              253,0     85144 25214105 /usr/lib64/python2.7/lib-dynload/cPickle.so
python  1459 root  mem    REG              253,0    402384  8431205 /usr/lib64/libpcre.so.1.2.0
python  1459 root  mem    REG              253,0    155752  8401283 /usr/lib64/libselinux.so.1
python  1459 root  mem    REG              253,0    109976  8453753 /usr/lib64/libresolv-2.17.so
python  1459 root  mem    REG              253,0     15688  8431853 /usr/lib64/libkeyutils.so.1.5
python  1459 root  mem    REG              253,0     67104  8453774 /usr/lib64/libkrb5support.so.0.1
python  1459 root  mem    REG              253,0     90248  8856160 /usr/lib64/libz.so.1.2.7
python  1459 root  mem    REG              253,0    210784  8431782 /usr/lib64/libk5crypto.so.3.1
python  1459 root  mem    REG              253,0     15856  8409570 /usr/lib64/libcom_err.so.2.1
python  1459 root  mem    REG              253,0    967776  8453772 /usr/lib64/libkrb5.so.3.3
python  1459 root  mem    REG              253,0    320720  8431744 /usr/lib64/libgssapi_krb5.so.2.2
python  1459 root  mem    REG              253,0   2521008  8410640 /usr/lib64/libcrypto.so.1.0.2k
python  1459 root  mem    REG              253,0    470360  8410643 /usr/lib64/libssl.so.1.0.2k
python  1459 root  mem    REG              253,0     95072 25403940 /usr/lib64/python2.7/lib-dynload/_ssl.so
python  1459 root  mem    REG              253,0     83872 25169643 /usr/lib64/python2.7/lib-dynload/_socketmodule.so
python  1459 root  mem    REG              253,0     25712 25169646 /usr/lib64/python2.7/lib-dynload/binascii.so
python  1459 root  mem    REG              253,0     39048 25169639 /usr/lib64/python2.7/lib-dynload/_json.so
python  1459 root  mem    REG              253,0     38952 25169644 /usr/lib64/python2.7/lib-dynload/_struct.so
python  1459 root  mem    REG              253,0     28696 25169658 /usr/lib64/python2.7/lib-dynload/stropmodule.so
python  1459 root  mem    REG              253,0     17008 25169636 /usr/lib64/python2.7/lib-dynload/_functoolsmodule.so
python  1459 root  mem    REG              253,0     22872 25339979 /usr/lib64/python2.7/lib-dynload/_heapq.so
python  1459 root  mem    REG              253,0     62016 25173693 /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so
python  1459 root  mem    REG              253,0     47624 25169650 /usr/lib64/python2.7/lib-dynload/operator.so
python  1459 root  mem    REG              253,0     33048 25321475 /usr/lib64/python2.7/lib-dynload/_collectionsmodule.so
python  1459 root  mem    REG              253,0     23912 25214107 /usr/lib64/python2.7/lib-dynload/cStringIO.so
python  1459 root  mem    REG              253,0     25016 25169661 /usr/lib64/python2.7/lib-dynload/timemodule.so
python  1459 root  mem    REG              253,0 106176928 17017165 /usr/lib/locale/locale-archive
python  1459 root  mem    REG              253,0     61560  8453743 /usr/lib64/libnss_files-2.17.so
python  1459 root  mem    REG              253,0   2156240  8401253 /usr/lib64/libc-2.17.so
python  1459 root  mem    REG              253,0   1136944  8453733 /usr/lib64/libm-2.17.so
python  1459 root  mem    REG              253,0     14424  8453759 /usr/lib64/libutil-2.17.so
python  1459 root  mem    REG              253,0     19248  8453731 /usr/lib64/libdl-2.17.so
python  1459 root  mem    REG              253,0    142144  8453751 /usr/lib64/libpthread-2.17.so
python  1459 root  mem    REG              253,0   1847768  8395510 /usr/lib64/libpython2.7.so.1.0
python  1459 root  mem    REG              253,0    163312  8393450 /usr/lib64/ld-2.17.so
python  1459 root    0r   CHR                1,3       0t0     1028 /dev/null
python  1459 root    1u  unix 0xffff969ea55e8880       0t0    36170 socket
python  1459 root    2u  unix 0xffff969ea55e8880       0t0    36170 socket
python  1459 root    3wW  REG               0,19         4    18823 /run/goferd.pid
python  1459 root    4u  unix 0xffff969bf37a5940       0t0    40858 socket
python  1459 root    5u  IPv4              35653       0t0      TCP server.domain.companynet.net:37316->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root    6u  IPv4              35655       0t0      TCP server.domain.companynet.net:37318->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root    7u  IPv4         1760413079       0t0      TCP server.domain.companynet.net:33302->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root    8u  IPv4         1760435018       0t0      TCP server.domain.companynet.net:33308->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root    9r   CHR                1,9       0t0     1033 /dev/urandom
python  1459 root   10u  IPv4         1760519157       0t0      TCP server.domain.companynet.net:speedtrace->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root   11u  IPv4         1760593723       0t0      TCP server.domain.companynet.net:33358->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root   12u  IPv4         1760593916       0t0      TCP server.domain.companynet.net:33376->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root   13u  IPv4         1760733109       0t0      TCP server.domain.companynet.net:33454->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root   14u  IPv4         1760832527       0t0      TCP server.domain.companynet.net:33486->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root 1021u  IPv4         2895569931       0t0      TCP server.domain.companynet.net:39920->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root 1022u  IPv4         2895534621       0t0      TCP server.domain.companynet.net:39926->DMZserver.companynet.net:5647 (CLOSE_WAIT)
python  1459 root 1023u  IPv4         2895546558       0t0      TCP server.domain.companynet.net:39936->DMZserver.companynet.net:5647 (CLOSE_WAIT)

Python is using almost 30% of SWAP file and it getting bigger and bigger. It could be ok but someone in my organization setup notification when SWAP file is larger then 40% and temporary I can't change it but I'm receiving all the time alerts from that server and it is annoying.

On that server is installed Bitbucket, PostgreSQL and Apache. I have tried swapoff -av && swapon -av but after a couple of hours it returned to previous size and again it's getting bigger and bigger.

Question is what for is use on that server Python process? I'm sure that Bitbucket or PostgreSQL is not using it but it looks that it some kind of network communication to external server (Bitbucket in our company is available also from external site) so it might be combined with Bitbucket.

What for is that process? Can I limit it somehow?

PS

Of course I could extend SWAP but for curiosity I need to know what for it is?

Edit#1 Output from command ps -aef --forest | grep python

root      1459     1  0 Jun17 ?        00:35:09 /usr/bin/python /usr/bin/goferd --foreground
root      1465     1  0 Jun17 ?        00:02:06 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
root     20739 22562  0 11:52 pts/0    00:00:00                          \_ grep --color=auto python
netdata   2528  2288  0 Jun17 ?        00:11:18  \_ /usr/bin/python /usr/libexec/netdata/plugins.d/python.d.plugin 1
splunk   31593     1  0 01:05 ?        00:00:03 python /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.py --fifo fifo2

Python is a (scripting) programming language. Therefore the important thing to look for is not python itself, but what program python is running. Have you looked for it with ps -ef | grep python? Since swap space is associated with high memory usage you may want to look at top. See how to display top results by memory usage in real time — Philip Couling, Jul 02 '20 at 09:34
Python is just an interpreter. The first step is to know what program this is. The fact that it happens to be written in Python is mostly irrelevant. lsof is not particularly useful information here. Start with ps ww 1459 — Gilles 'SO- stop being evil', Jul 02 '20 at 09:35
For ps ww 1459 output is: 1459 ? Ssl 35:08 /usr/bin/python /usr/bin/goferd --foreground — KUE, Jul 02 '20 at 09:47

score 1 · Accepted Answer · answered Jul 02 '20 at 20:56

1

Known bug: "Memory leak in goferd on clients connected to Red Hat Satellite 6.2.x when qrouterd is unavailable".

answered Jul 02 '20 at 20:56

Paul_Pedant

8,679

Python consume too much SWAP

1 Answers1