3

I would like to have a script that would prepend each line of a stdin with information how long it took to generate it. Basically for input:

foo
bar
baz

I would like to have

0 foo
10 bar
5 baz

Where 10 is 10 seconds passed between printing foo and printing bar, similar for 5, it took 5 seconds after printing bar to print baz.

I know there is a utility ts that shows timestamps and I know about https://github.com/paypal/gnomon, but I would prefer not to use javascript to do that.

Is there a standard tool for that or should I use awk and do processing?

Krzysztof Krasoń
  • 195

5 Answers5

7

Let's suppose that the script generating the output is called generate. Then, to display the number of seconds it takes for it to generate each line:

$ generate | ( t0=$(date +%s); while read -r line; do t1=$(date +%s); echo " $((t1-t0)) $line"; t0=$t1; done )
 2 foo
 0 foo
 5 foo
 3 foo

The same commands spread out over multiple lines looks like:

generate | ( t0=$(date +%s)
    while read -r line
    do
        t1=$(date +%s)
        echo " $((t1-t0)) $line"
        t0=$t1
    done
    )

Alternatively, for convenience, we can define a shell function that contains this code:

timer() { t0=$(date +%s); while read -r line; do t1=$(date +%s); echo " $((t1-t0)) $line"; t0=$t1; done; }

We can use this function as follows:

$ generate | timer
 0 foo
 2 foo
 4 foo
 3 foo

How it works

  • t0=$(date +%s)

    This captures the current time at the start of the script in seconds-since-epoch.

  • while read -r line; do

    This starts a loop which reads from standard input

  • t1=$(date +%s)

    This captures the time in seconds-since-epoch at which the current line was captured.

  • echo " $((t1-t0)) $line"

    This prints out the time in seconds that it took for the current line.

  • t0=$t1

    This updates t0 for the next line.

  • done

    This signals the end of the while loop.

John1024
  • 74,655
5
trickle () {
    awk 'BEGIN { t = systime(); OFS="\t" }
               { print systime() - t, $0 }
               { t = systime()           }'
}

This little shell function simply wraps an awk script (compatible with both GNU awk, and mawk -Wi, but not with BSD awk due to it missing systime()). It will output each line of input prefixed with the number of whole seconds since the last line of output. The first line will be prefixed with a zero.

Testing:

$ { echo hello; sleep 2; echo world } | trickle
0       hello
2       world

The awk script could obviously be run stand-alone as well:

$ some_command | awk 'BEGIN{OFS="\t";t=systime()}{print systime()-t,$0;t=systime()}

E.g.

$ { echo hello; sleep 2; echo world; } | awk 'BEGIN{OFS="\t";t=systime()}{print systime()-t,$0;t=systime()}'
0       hello
2       world
Kusalananda
  • 333,661
3

Well, moreutils ts seems to be a perfect tool for that:

for seconds since the start:

$ seq 5 | pv -qL 1 | ts -s %s
2 1
4 2
6 3
8 4
10 5

Or with more precision:

$ seq 5 | pv -qL 1 | ts -s %.s
1.949228 1
3.933483 2
5.917923 3
7.902231 4
9.886618 5

Or with zsh or ksh93 ($SECONDS is integer by default but you can make it float with typeset -F; pdksh and bash have $SECONDS as well, but integer only):

$ typeset -F SECONDS=0; seq 5 | pv -qL 1 | while IFS= read -r line; do
   printf '%.3f %s\n' "$SECONDS" "$line"; done
1.987 1
3.972 2
5.956 3
7.941 4
9.925 5

for seconds between lines

$ seq 5 | pv -qL 1 | ts -i %.s
1.947814 1
1.984442 2
1.984456 3
1.984151 4
1.984195 5

zsh/ksh93:

$ typeset -F SECONDS; seq 5 | pv -qL 1 | while SECONDS=0; IFS= read -r line; do
  printf '%.6f %s\n' "$SECONDS" "$line"; done
1.985298 1
1.984322 2
1.984212 3
1.984240 4
1.984441 5
Stéphane Chazelas
  • 544,893
2

You can use time (note: a separate binary, not the shell built-in) and combine it with a while loop. To get the execution time for each command:

while read -r line; do
    /usr/bin/time -f "%e %C" $line 2>&1
done

Example output:

0.1 sleep 0.1
0.2 sleep 0.2
1.0 sleep 1

In case you want the total execution time from the beginning, you need to sum the execution time for each step. Precision for %e format is 1/100th of second, so you can't use shell integer arithmetic directly (depending on the shell you using).

total=0.0

while read -r line; do
    t=$(/usr/bin/time -f "%e" $line 2>&1)
    total=$(awk "BEGIN{print $total+$t}")
    echo "$total $line"
done

Example output:

0.1 sleep 0.1
0.3 sleep 0.2
1.3 sleep 1
sebasth
  • 14,872
  • It doesn't look correct, I want o have information how long it took to generate the line, not a time from the beginning. In your case it should be 0.2 sleep 0.2 0.2 sleep 0.2 1.0 sleep 1 – Krzysztof Krasoń Sep 09 '17 at 06:12
  • I've edited the answer, the first example works as you described. – sebasth Sep 09 '17 at 09:32
1

First, create a simple function leveraging the bash builtin SECONDS. Notice we initialize it to zero on each invocation.

_time() { SECONDS=0; out="$($@)"; echo "[${SECONDS}] ${out:-no output}"; }

Next, use this _time function to run your command(s) from a list of commands could be a file of them, or whatever.

while read cmd; do _time $cmd; done < <(echo -e "sleep 5\nsleep 2\nsleep 1")

Here, sleep has no output, but your commands likely will. This is simply to show the use of SECONDS as it pertains to this problem. This example will output the default output of no output:

[5] no output
[2] no output
[1] no output
chris
  • 21