3

My problem

I need to find out how much resources are required by a bash script I wrote. I'm mainly interested in the following:

  • RAM

  • peak disk usage (disk space)

  • CPU time: while here I'm not sure to what this exactly refers to the execution time on whole CPU (including physical + virtual cores?) or just the cores which where used. For example, how do I know how much CPU time is required on a CPU with a different number of cores?

  • used number of cores

The reason for this is, that I will submit jobs to a cluster. Where each job executes my script with slightly changed parameters.

My plan is the following, I execute the script on my pc to get the peak values of the above resources and then make sure that only cluster nodes are used which have the required amount of resources ready.

Or even better during script execution I write the desired stuff to some file.

Attempted solution

What I did so far:

RAM: measured the peak RAM of the called program in my script from which I guessed it has the maximum RAM usage. (This was a Mathematica Script where I used the command MaxMemoryUsed[] which I later extract from the log file)

Disk Space: I basically sum over all files / folders using du -sb. Moreover, I also use df before and after the script.

CPU time: at the beginning of the script I set SECONDS=0 and echo ${SECONDS} at the end (Since the script is the only thing I am running this should correspond to the CPU time?!). I also used date to get a rough estimation.

I'm thankful for all hints, comments and pointers to tools and possible solutions to my problem.

Edit

I also heard about valgrind. Does anybody have some experience with that. Is it possible to use this for my purposes?

Stefan
  • 135

1 Answers1

7

You can do this by running the command inside GNU time. By default, time shows you the real (wall clock), user (CPU-seconds used in user mode), and sys (CPU-seconds used in kernel mode) data items. However, you can also ask it to measure other things, such as RAM and disk usage:

/usr/bin/time -f "File system outputs: %O\nMaximum RSS size: %M\nCPU percentage used: %P" <command>

where <command> is replaced by the command you wish to run. The output will be something like:

File system outputs: 18992
Maximum RSS size: 40056
CPU percentage used: 367%

where "CPU percentage used" is a percentage and shows here that 3.6 cores were used, "Maximum RSS size" is as close as it gets to "maximum memory used" and is expressed in kilobytes, and "File system outputs" is expressed in number of operations (i.e., it does not say how much data is written). The du and df commands you gave should help there.

Note: you need to use /usr/bin/time rather than just time, as many shells have that as a builtin, which doesn't necessarily support the -f option.

For more information, see man time

Wouter Verhelst
  • 9,331
  • Hmm, /usr/bin/time does not exist and as you already stated time alone have no option -f (only -p). In addition, I'm not absolutely sure how to call my commands using the /usr/bin/time command you stated (once I've found it if there is one at my machine?!). Since all of them receive variables + redirect stdout and stderr to files. Do I just have to use a local command group using (<command>) to be able to redirect the output of \usr\bin\time also to a file by just adding > time.out afterwards – Stefan Dec 21 '16 at 13:50
  • On my Debian, there is a time package which is not installed by default. You'll need to install that manually. If you're not running Debian, and your distribution does not have the time package, you can download it from the GNU website and compile it yourself: http://directory.fsf.org/wiki/Time – Wouter Verhelst Dec 21 '16 at 16:48
  • /usr/bin/time works now as expected, but I needed one more step in my execution line in order to get it working (I now submit an intermediate script to the cluster which just executes usr/bin/time on the main script that does all the work). However, in order to keep better track of where most of the resources are used, I'm wondering if and how it is possible to apply /usr/bin/time on a bash function? – Stefan Jan 13 '17 at 08:19