47

Let's say I'm running a script (e.g. in Python). In order to find out how long the program took, one would run

time python script1.py

Is there a command which keeps track of how much RAM was used as the script was running? In order to find how much RAM is available, one could use free, but this command doesn't fit the task above.

ShanZhengYang
  • 1,653
  • 2
    https://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix-process – muru Jul 07 '17 at 04:49
  • 2
    Also related, if not a duplicate: https://unix.stackexchange.com/q/164653/85039 – Sergiy Kolodyazhnyy Jul 07 '17 at 08:33
  • Why exactly do you ask? Why do you care more about physical RAM than about virtual address space consumed by your python process? – Basile Starynkevitch Jul 08 '17 at 06:34
  • @BasileStarynkevitch There's nothing in the question that indicates that the user only wants to know about physical RAM. – pipe Jul 08 '17 at 07:18
  • 1
    RAM is only physical. It is a concrete device. Virtual memory (or just memory) is an abstraction. There is no such thing as "non-physical" RAM, and we generally don't care about RAM, we care about (virtual) memory. OP could have asked "how much memory was used" – Basile Starynkevitch Jul 08 '17 at 07:21
  • Also, knowing how much physical ram per instance a process uses, both peak and average, can be quite important when sizing servers and batch processing computers (if the processes are not I/O intensive - otherwise, you want to buy as much RAM as you can to get plenty of buffering anyway). – rackandboneman Jul 08 '17 at 18:15

3 Answers3

63

The time(1) command (you may need to install it -perhaps as the time package-, it should be in /usr/bin/time) accepts many arguments, including a format string (with -f or --format) which understands (among others)

  %M     Maximum resident set size of the process during its lifetime,

          in Kbytes.

  %K     Average total (data+stack+text) memory use of the process, in

          Kbytes.

Don't confuse the /usr/bin/time command with the time bash builtin. You may need to type the full file path /usr/bin/time (to ask your shell to run the command not the builtin) or type command time or \time (thanks to Toby Speight & to Arrow for their comments).

So you might try (RSS being the resident set size)

/usr/bin/time -f "mem=%K RSS=%M elapsed=%E cpu.sys=%S user=%U" python script1.py

You could also try

/usr/bin/time --verbose  python script1.py

You are asking:

how much RAM was used as the script was running?

and this shows a misconception from your part. Application programs running on Linux (or any modern multi-process operating system) are using virtual memory, and each process (including the python process running your script) has its own virtual address space. A process don't run directly in physical RAM, but has its own virtual address space (and runs in it), and the kernel implements virtual memory by sophisticated demand-paging using lazy copy-on-write techniques and configures the MMU. The RAM is a physical device and resource used -and managed internally by the kernel- to implement virtual memory (read also about the page cache and about thrashing).

You may want to spend several days understanding more about operating systems. I recommend reading Operating Systems : Three Easy Pieces which is a freely downloadable book. The RAM is used by the entire operating system (not -directly- by individual processes) and the actual pages in RAM for a given process can vary during time (and could be somehow shared with other processes). Hence the RAM consumption of a given process is not well defined since it is constantly changing (you may want its average, or its peak value, etc...), and likewise for the size of its virtual address space.

You could also use (especially if your script runs for several seconds) the top(1) utility (perhaps in some other terminal), or ps(1) or pmap(1) -maybe using watch(1) to repeat that ps or pmap command. You could even use directly /proc/ (see proc(5)...) perhaps as watch cat /proc/$(pidof python)/status or /proc/$(pidof python)/stat or /proc/$(pidof python)/maps etc...

But RAM usage (by the kernel for some process) is widely varying with time for a given process (and even its virtual address space is changing, e.g. by calls to mmap(2) and munmap used by ld-linux(8), dlopen(3), malloc(3) & free and many other functions needed to your Python interpreter...).

You could also use strace(1) to understand the system calls done by Python for your script (so you would understand how it uses mmap & munmap and other syscalls(2)). You might restrict strace with -e trace=%memory or -e trace=memory to get only memory (i.e. virtual address space) related system calls.

BTW, the tracemalloc Python feature could be also useful.

I guess that you only care about virtual memory, that is about virtual address space (but not about RAM), used by the Python interpreter to run your Python script. And that is changing during execution of the process. The RSS (or the maximal peak size of the virtual address space) could actually be more useful to know.

See also LinuxAteMyRAM.

vfclists
  • 7,531
  • 14
  • 53
  • 79
Basile Starynkevitch
  • 10,561
  • That's great! Thank you, +1. Can you explain what they mean with "resident set size"? does this include stack and heap? – Philippos Jul 07 '17 at 05:56
  • Look on wikipedia (because explaining RSS could take many pages). Better yet, spend a few days reading a good book like Operating Systems: Three Easy Pieces which is freely downloadable. – Basile Starynkevitch Jul 07 '17 at 07:14
  • 2
    You expect me to spend days on reading a book with probably 99% already known to me to find the interpretation of a term? A simple "yes" or now" could have save me a lot of time. )-: – Philippos Jul 07 '17 at 07:38
  • 2
    But the notion of RSS (related to that of virtual memory) is not simple to explain. Perhaps the wikipage could be enough to you. My point is that your naive question (how much RAM does a python script spends) has no precise meaning, because processes don't use RAM (but virtual address space, which is not the same). What exactly happens requires indeed at least a whole book (and perhaps more) to be explained. – Basile Starynkevitch Jul 07 '17 at 07:48
  • 1
    It's neither my question nor am I naive. I've done a lot of things like MMU configuration switching depending on tasks for an RTOS and dynamic loading of libraries in bare metal systems, I just don't know how this term was used in the man page. As I see now, it seems to be a fixed term in the unix world. Most of it is less relevant in my embedded one-process without swap world. – Philippos Jul 07 '17 at 07:58
  • And if did know 99% of the textbook on OSes, you won't ask about RSS (that notion should be known to you). And no, RSS is not a Unix specific term (IIRC, IBM OS/VS2 used it in 1970s and VAX/VMS in the 1980s). – Basile Starynkevitch Jul 07 '17 at 07:58
  • 7
    I'm not interested in your style of discussion if you prefer to become personal when you can't distinguish between concepts and terms. – Philippos Jul 07 '17 at 08:08
  • 4
    You don't need the full path - command time will tell Bash to ignore the builtin. That's useful if you can't remember which bin directory holds it. – Toby Speight Jul 07 '17 at 08:51
  • 4
    @TobySpeight Or simpler: \time will tell a shell to search in the PATH. A backslash will not match exactly the time builtin name, so a search will be initiated in the PATH. –  Jul 07 '17 at 14:09
  • Another naïve, ill-informed question: RE: %K and %M Is it possible to see the outputs in units other than KB? For instance, GB? From the manual, I don't quite see this option – ShanZhengYang Jul 07 '17 at 16:50
  • 1
    @ShanZhengYang No, not directly from time. The manual entry for K and M are pretty clear about it IMnshO: M … … , in Kilobytes. –  Jul 07 '17 at 17:48
  • 1
    A process has a larger allocated address space than it actually uses in physical/virtual memory, since shared libraries (but not the data these allocate!) only count once if they are loaded by more than one process. On top of that, there is mmap(). – rackandboneman Jul 07 '17 at 18:56
  • @Arrow Apologies, my reading comprehension is worse than my naïveté and ignorance. – ShanZhengYang Jul 07 '17 at 20:03
  • @Basile Starynkevitch: If you think processes don't use RAM, write a program that needs say 4.1 Gbytes, and run it on a 4 GByte machine. – jamesqf Jul 08 '17 at 05:21
  • 1
    Processes are using virtual address space, which is provided by the kernel. The kernel uses RAM to provide that. The programs don't use any RAM directly (there is no way to directly use RAM for a Linux application program, outside of mlock, /dev/mem, and a few other tricks). What you suggest is thrashing that my answer did tell about – Basile Starynkevitch Jul 08 '17 at 05:25
  • The Wikipedia explanation is only two sentences long: https://en.wikipedia.org/wiki/Resident_set_size . IIRC RSS includes executable code, heap, stack, 100% of pages shared with other processes, but excludes non-resident pages such as zero-pages or pages that have been swapped out. – gmatht Jul 08 '17 at 05:42
  • My feeling is that RSS is a complex and delicate notion, that I am not able to explain completely in a few words. So I believe you need to read an entire book (mentioned in my answer) to grasp completely that notion. Notice that for a given process the set of pages in RAM for it is constantly changing! – Basile Starynkevitch Jul 08 '17 at 05:43
  • Is the . as in .user in the example command necessary? I edited it out. – vfclists Oct 30 '21 at 22:11
17

You can also use the legendary valgrind, although you may need to install it from your package manager.

$ valgrind c-projects/objtest/bin/objtest 
==6543== Memcheck, a memory error detector
==6543== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==6543== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==6543== Command: c-projects/objtest/bin/objtest
==6543== 
|ERROR|array:189: array_delete: delete index 0 but the highest is -1 (delete from empty array): index out of bounds
==6543== 
==6543== HEAP SUMMARY:
==6543==     in use at exit: 480 bytes in 20 blocks
==6543==   total heap usage: 7,390 allocs, 7,370 frees, 256,217 bytes allocated
==6543== 
==6543== LEAK SUMMARY:
==6543==    definitely lost: 96 bytes in 4 blocks
==6543==    indirectly lost: 384 bytes in 16 blocks
==6543==      possibly lost: 0 bytes in 0 blocks
==6543==    still reachable: 0 bytes in 0 blocks
==6543==         suppressed: 0 bytes in 0 blocks
==6543== Rerun with --leak-check=full to see details of leaked memory
==6543== 
==6543== For counts of detected and suppressed errors, rerun with: -v
==6543== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If you are a power user and want fancy graphs, you can use ms_print and a command like:

valgrind --tool=massif --pages-as-heap=yes --massif-out-file=massif.out ./test.sh; grep mem_heap_B massif.out | sed -e 's/mem_heap_B=\(.*\)/\1/' | sort -g | tail -n 1

see Peak memory usage of a linux/unix process.

cat
  • 3,468
  • 4
  • 23
  • 51
  • 2
    This is a good suggestion - it may be worth pointing out that you can't tell the difference between a large amount of memory allocated for a long time and many smaller short-term allocations. It's hard to know whether the questioner is also interested in other memory usage (stack, mmapped files, text segments etc). – Toby Speight Jul 07 '17 at 08:55
  • @TobySpeight If you look at the linked answer, you can actually get histogram output of much more specific data – cat Jul 07 '17 at 15:27
  • Consider editing to add that comment about the link, so that others know why they might want to follow it. I haven't used Massif very much, but did think it was probably a better choice than Memcheck here - thanks for confirming that! – Toby Speight Jul 10 '17 at 08:07
2

You can use pmap command to view the amount of memory used by a process. In your case you need to give the PID of the script as input to the pmap command like

pmap $(ps -ef | grep **<SCRIPT NAME>** | grep -v grep | awk '{print $2}')

upkar
  • 326
  • 4
    That only gives a snapshot, not the max (or even average) over the life of the process. Also consider pgrep https://unix.stackexchange.com/questions/74185/how-can-i-prevent-grep-from-showing-up-in-ps-results – dave_thompson_085 Jul 07 '17 at 07:02