31

I want to monitor memory usage of a process, and I want this data to be logged. Does such a tool exist?

tshepang
  • 65,642

7 Answers7

30

Occasionally when the need arises I just do:

$ top -d 1 -b |grep <process> >>somefile

It's not an elegant solution, but gets the job done if you want the quick crude value to verify your hypothesis.

slm
  • 369,824
Sridhar Iyer
  • 411
  • 3
    I think it is elegant in it's simplicity. You may want to do grep --line-buffered <process> >>somefile to force grep to output each line without buffering – Ott Toomet Oct 07 '17 at 21:04
  • 1
    An explanation of the top arguments: -b sets top to batch mode, suitable for outputting to other processes, and -d sets a delay in seconds for each update. Altogether this command outputs lines from top every 1 second and appends to somefile. – congusbongus Nov 04 '20 at 23:00
  • 1
    This doesn't seem to append anything to somefile for me. I am using top -d 1 -b | grep 2597 >>somefile, somefile gets created, but is always empty. Any ideas on what's wrong? – Dahn Feb 25 '21 at 18:19
13

I have written a script to do exactly this. It basically samples ps at specific intervals, to build up a profile of a particular process. The process can be launched by the monitoring tool itself, or it can be an independent process (specified by pid or command pattern).

Sibren
  • 103
Jeet
  • 246
8

sar (System Activity Reporter) from the sysstat package is your friend in case like these.

Another way would be monitoring combined with historical data, e.g. Munin, pnp4nagios, rrdtools, ...

Christian
  • 1,591
4

Besides the aforementioned sar, I'd recommend atop. It saves a binary log that you can peruse afterwards, and besides memory saves a lot of other information.

rsuarez
  • 902
2

You could try Valgrind.

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache and branch-prediction profiler, and a heap profiler.

andcoz
  • 17,130
2

I want to monitor memory usage of a process, and I want this data to be logged. Does such a tool exist?

Yes, can achieve exactly this, a bit more with Procpath (author here).

In the simplest case you point it to the PID of interest and let it record its Procfs metrics including Resident Set Size (RSS). By default it makes a recording per 10 seconds (for the target process, its ancestors and descendants).

$ procpath record -d tgt.sqlite -p $PID

When you think it has recorded enough datapoints, Ctrl+C it and this will give you the recordings.

$ sqlite3 tgt.sqlite "SELECT ts, stat_rss * 4 rss_kib FROM record WHERE stat_pid = $PID"
1635349330.11141|478460
1635349340.12125|476068
1635349350.13124|480232
1635349360.14117|483416
...

Using a SQLite GUI can be a more convenient to explore and query the database (e.g. apt install sqlitebrowser).

Procpath can also visualise RSS of the target process into an SVG out of the box like this:

$ procpath plot -d tgt.sqlite -f tgt.svg -q rss -p $PID

resident set size plot

If you're running Kernel 4.14+ you can get more advanced Linux memory usage metrics like Proportional Set Size (PSS), Unique Set Size (USS), proportional swap usage and others (see the documentation for what you can get from /proc/{pid}/smaps_rollup):

procpath record -d tgt.sqlite -p $PID -f stat,cmdline,smaps_rollup
saaj
  • 2,438
1

I like Sridhar's simple answer but I rolled my own before I tried his:

#! /usr/bin/python3

import json, psutil, datetime, sys, time

while True:
  print(json.dumps((datetime.datetime.now().isoformat(),
                    psutil.Process(int(sys.argv[1])).memory_info()._asdict())))
  time.sleep(1)

It's basically only useful if you want structured output. Note: _asdict() was broken in some versions of Python 3.5; it works again in Python 3.6.9 though (as well as in Python 2.7.17).

Output looks like:

["2019-03-19T11:21:53.784670", {"rss": 220389376, "vms": 538984448, "shared": 15724544, "text": 303104, "lib": 0, "data": 221364224, "dirty": 0}]
["2019-03-19T11:21:54.786136", {"rss": 220438528, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]
["2019-03-19T11:21:55.787555", {"rss": 220495872, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]
["2019-03-19T11:21:56.788754", {"rss": 220528640, "vms": 539119616, "shared": 15724544, "text": 303104, "lib": 0, "data": 221499392, "dirty": 0}]

For me, it was important to have structured output so that I could more easily consume it for analysis.

Robert Fleming
  • 457