Stable timing for shell scripts?

Question

The run times of scripts vary quite much. I would like to find similar time -function as Matlab's timeit, described here.

For instance, I am evaluating these commands here about counting quickly matches, running time LC_ALL=C grep -ao CDA r328_0002.raw | wc -l in loop

---------------------------------------------
Events      real        user        sys
----------- ----------- ----------- ---------
40          0m0.044s    0m0.042s    0m0.005s
40          0m0.064s    0m0.062s    0m0.005s
40          0m0.046s    0m0.044s    0m0.005s
40          0m0.043s    0m0.042s    0m0.005s
40          0m0.047s    0m0.044s    0m0.005s
---------------------------------------------

Table: Events when Macbook Air 2013-Mid in Power Supply. 

---------------------------------------------
Events      real        user        sys
----------- ----------- ----------- ---------
40          0m0.056s    0m0.041s    0m0.011s
40          0m0.060s    0m0.047s    0m0.008s
40          0m0.041s    0m0.039s    0m0.006s
40          0m0.046s    0m0.044s    0m0.006s
40          0m0.047s    0m0.045s    0m0.006s
---------------------------------------------

Table: Events when Macbook Air in Battery Supply, 6h later.  

where you see the real-time fluctuates from 0.044s to 0.064s, the user-time from 0.042s to 0.062s, while the sys-time keeps rather stable at 0.005s. My idea in timing

• iterate the command first 1k before timing
• do timing 10 times and take the average and standard deviation

Stout to /dev/nul

This idea is in lcd047's comment, running time LC_ALL=C ggrep -ao CDA r328_0002.raw >/dev/null in loop

--------------------------------------------
real            user            sys
--------------  --------------  ------------
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.008s        0m0.003s        0m0.003s
0m0.006s        0m0.003s        0m0.002s
0m0.005s        0m0.002s        0m0.002s
0m0.006s        0m0.002s        0m0.002s
0m0.009s        0m0.003s        0m0.003s
0m0.007s        0m0.003s        0m0.003s
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.008s        0m0.003s        0m0.003s
--------------------------------------------

Table: Events when Macbook Air 2013-Mid in Battery Supply. 

I think these times can be even more improved when keeping the laptop in power supply and keeping less programs on.

How can you time stably shell scripts?

Answer 1

The answer is: You can not! Linux is no real time system. The idea of UNIX and therefore Linux, too, is to provide minimum answer times, while the system is shared between multiple users and system processes. Depending on when you start the command, you might have to wait for a important system process to give you your share of processor time. Further the file system might buffer your file you read from the disk, but eventually these file system buffer fail to load the data from cache, when another process on your system occupies that. Generally the time a process needs in a Linux system depends on the surrounding entropy of the machine, where ever it is located in time and space in the universe.

You will need a real time system and specially real time tuned commands and a specific amount of resources reserved for you only. You might get somewhere near that with new CGROUP features from newer kernels, where you can reserve a processor, a part of the memory and an own reflection of the underlying file system for your environment.

One of the major timing problems in your example is the way grep and wc reads the inputs. You might be able to make the timing more stable when you copy your file into a ramfs and work there.

Answer 2

To begin with, you need to understand that shell scripts are just a convenient shorthand for typing the commands yourself. Every time you or a script invokes grep, you're not calling a built-in primitive; you're asking the shell to hunt down a program called grep and start it as a new process. Creation of a new process and running its first bit of code is probably the least-predictable thing you can do in any environment that isn't tuned for predictability. That's going to be the case for any computer running a non-real-time operating system. The details of why this is would be fodder for half of a semester class in operating systems, but I can give you an example or two that illustrate why your efforts to precisely measure full program executions aren't going to yield the consistent results you think they should.

The first things most programs do when first given the CPU is generate a page fault so the first page's worth of code can be loaded. If a dozen other programs are doing I/O on the same device where the code lives, how long your program's pages will take to load depends on how far back the requests are in the device's queue. You may think your test program is running solo, but I'd bet dollars to doughnuts it isn't. (It's also worth mentioning that grep is an I/O-heavy program, so how long it takes to read the input varies for many of the same reasons.)

Many operating systems take steps to avoid having redundant copies of the same code resident at the same time as a way to cut down on memory consumption and increase performance. This means that if you start grep and there's another grep running with its first page already resident, the aforementioned page fault never happens and all of that effort is skipped. This decreases the wall-clock run time.

By the time you get around to doing a timeit in MATLAB, the MATLAB process is already running and it's likely jumped through the necessary hoops to load your function before invoking it repeatedly. Invocation happens fairly quickly because it's just calls internally. While there are also a lot of factors that would affect how long it takes timeit to run, they'd apply equally to anything else.

That said, I think the real reason for what you're seeing is that the comparison isn't apples-to-apples.

Unix's time(1) runs the program under test exactly once where MATLAB's timeit times the function you're testing numerous times and returns the median of the results. The documentation suggests that timeit and the tic and toc functions shouldn't be used together, which hints that the latter are used by the former. The docs for tic and toc recommend that if you have code that runs in anything less than 0.1 seconds, you run it many times and calculate an average. What I take from this is that MATLAB's timing is two orders of magnitude less precise than that of time(1) and is intended more for checking how much time you're able to shave off of long-running functions. That and the combination of averaging and a median will, given enough samples, push a lot of variation into a fairly consistent result.