Moving average on a log file with awk or other unix utilities?

Question

Scenario:
I have a log file that has a few number of entry "classes", like this:

R0 dx=0.00500 rb=0.00000 sn=1 3145.88 2.59 0.08 se=21315 id=16190  
R0 dx=0.00300 rb=-1.00000 sn=1 3150.40 2.38 0.05 se=21316 id=16191  
R1 dx=-0.00500 rb=1.00000 sn=-1 3155.14 2.54 0.05 se=21317 id=16192  
R1 dx=-0.02000 rb=-1.00000 sn=-1 3157.73 2.48 0.10 se=21318 id=16193  
R0 dx=-0.02000 rb=0.00000 sn=-1 3160.59 2.74 0.08 se=21319 id=16194   
R1 dx=0.00500 rb=1.00000 sn=1 3165.18 2.43 0.10 se=21320 id=16195     
R0 dx=0.00100 rb=-1.00000 sn=1 3167.84 2.53 0.05 se=21321 id=16196    
R3 dx=0.00100 rb=1.00000 sn=1 3170.11 0.10 0.10 se=21322 id=16197

I am monitoring this logfile with a tail -F (Is there a way to make tail -F beep?)

My question is: Is there a way to calculate the sliding (moving) ratio of "number of R1 lines to number of R0 lines" to give you an example? In addition to printing this ratio I need to be able to pass it to some other tool like gnuplot to plot it (although this is might be a bit of a stretch).

Obviously this is a rather trivial thing to do in Python or MATLAB or Octave but I am very eager to learn to do it in the shell, I think the tricky part is to pass the values to a plotting utility to update a plot.

angus · Answer 1 · 2012-01-18T19:03:02.603

2

Filter your log through this:

awk '/^R1/ { r1++ } ; /^R0/ { r0++ } ; r0 > 0 {  print r1/r0 }'

That should output the "running ratio" for each line after the first R0.

And this is for the sliding ratio (sorry about that, I had a wrong idea of what "ratio" meant):

BEGIN { winsize=10; h=t=r0=r1=0 }
/^R0/ { r0++ }
/^R1/ { r1++ }
      { print r1 ":" r0 ; buf[h++] = $1 }
h - t >= winsize { r = buf[t]; delete buf[t++] }                   
r ~ /R1/ { r1-- }
r ~ /R0/ { r0-- }

The variable winsize holds the window size. The regexes are superfluous here, but they save keystrokes. The tests could have been $1 == "R0" and r == "R0".

edited Jan 18 '12 at 19:03

answered Jan 17 '12 at 20:02

angus

12,321
3
45
40

But that's not a moving ratio. – Dennis Williamson Jan 18 '12 at 17:47
@DennisWilliamson: Yeah, for some reason I thought "ratio" meant "quotient", and a "sliding quotient" of course doesn't make sense, so I aimed for the next best thing. I now see a ratio can be expressed as m:n; I updated my answer. – angus Jan 18 '12 at 18:27
ratio == quotient, from Wiktionary: "(arithmetic) The relative magnitudes of two quantities (usually expressed as a quotient)." – Dennis Williamson Jan 18 '12 at 20:25
OK, then I was right. In that case, the script just shows how many R0 and R1 were in the last 10 lines. Maybe that's what the OP wanted. A ratio is obviously impossible to calculate when one of the quantities is 0, which should occur pretty often in a n-line window. – angus Jan 18 '12 at 21:26

score 2 · Accepted Answer · answered Jan 17 '12 at 23:25

Make tail -F beep once for every line:

bel=`echo foo | tr -c -s '\007' '\007'`
tail -F file | sed "s/\$/$bel/"

As for using the shell to compute a moving average, here's a bash script that tracks the number of R0 and R1 lines within a moving window of size $windowsize. Tracking variables are r0sum and r1sum.

windowsize=10
declare -ai isr0line isr1line r0sum r1sum i
for ((i=0; $i<$windowsize;i+=1)) ; do isr0line[$i]=0; isr1line[$i]=0; done
i=0
while read line
do
   r0sum=$(($r0sum - ${isr0line[$i]}))
   r1sum=$(($r1sum - ${isr1line[$i]}))
   case "$line" in
      R0*) isr0line[$i]=1; isr1line[$i]=0; ;;
      R1*) isr1line[$i]=1; isr0line[$i]=0; ;;
        *) isr0line[$i]=0; isr1line[$i]=0; ;;
   esac
   r0sum=$(($r0sum + ${isr0line[$i]}))
   r1sum=$(($r1sum + ${isr1line[$i]}))
   echo "R0 lines $r0sum  R1 lines $r1sum"
   i=$((($i + 1) % $windowsize))
done

What is the purpose of the echo and tr in the assignment of bel? In Bash (and some other shells) you can do bel=$'\007'. Another method: bel=$(echo -n '\007') — Dennis Williamson, Jan 18 '12 at 17:56
Habit. I normally write for sh since that's all there was when I learned Unix. I took on this exercise to prod myself to learn a few more bash features. I didn't know the shell handled octal escapes in strings directly now. — Kyle Jones, Jan 18 '12 at 18:29

Moving average on a log file with awk or other unix utilities?

2 Answers2