43

I want to do some simple computation of the number of lines per minute added to a log file. I also want to store the count for each second.

What I need is the output of the following command as a list which will be updated every second:

watch -n1 'wc -l my.log'

How can I output the 'update' of the 'watch' command as a list?

slm
  • 369,824
JohnJohnGa
  • 531

6 Answers6

51

You can use the -t switch to watch which causes it not to print header. However, that will still clear the screen so you might be better off with a simple shell loop:

while sleep 1; do
    wc -l my.log
done

One of the advantages is, that you can easily add other commands (e.g. date) and/or pipe the output through sed to reformat it. By the way, if you swap sleep 1 with wc in the loop, it will automatically terminate on errors.

Stéphane Chazelas
  • 544,893
peterph
  • 30,838
  • However note that it will not do it exactly every second (with zsh or ksh93, you could adjust the sleeping time so as to account for the drift incurred by running the commands in the loop, though) – Stéphane Chazelas Nov 19 '12 at 11:51
  • 2
    @StephaneChazelas neither will it with wait - just try watch -n 1 "sleep 5". – peterph Nov 19 '12 at 12:26
  • Indeed (I've checked both procps and busybox implementations). I thought that was the only thing watch was useful for, but it doesn't give you even that so your solution is as good as a watch based one, but neither answers the "rate at which a log file grows" question with great accuracy. – Stéphane Chazelas Nov 19 '12 at 13:22
  • Well, if lines per minute are the ultimate goal, then sampling it once every 10 seconds is more than enough imho - so the overhead is not that terrible (unless the file grows really big, of course). And actually from within the loop, one can print timing information (even both before and after the command finishes if necessary), and then the accuracy can get better (orders of magnitude) even for bigger files. – peterph Nov 19 '12 at 13:28
  • 2
    @peterph watch has a -p option that will do that right, if at all possible (obviously, you can't do a command that takes 5 seconds every 1 second, if you're not allowed multiple simultaneous). I know, I wrote it :-P – derobert Jun 19 '14 at 17:46
  • @derobert you mean watch from procps-ng? For the "canonical" one from the procps definitely doesn't have this option. – peterph Jun 20 '14 at 07:19
  • @peterph Yeah, I sent it to Debian in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=183486 and Debian is using procps-ng. AFAIK, original procps upstream is unmaintained. – derobert Jun 20 '14 at 07:25
  • @derobert oh. Well, a quick glance at the procps CVS reveals how active the codebase has been in the last decade or so. I guess that's why it got forked into procps-ng (including your patch). – peterph Jun 20 '14 at 07:33
  • Another benefit of this approach is that watch only supports intervals down to 0.1 seconds, while sleep supports "floating point numbers" as intervals (e.g. sleep 0.0001 works just fine). – rinogo Oct 16 '20 at 01:20
14

An old question, but I just found a very easy answer:

watch -n1 'wc -l my.log | tee -a statistics.log'

This will execute your wc each second, add its output to the statistics.log file, and also show it on the screen.
So, you'll end up with a file filled with numbers, representing the successive number of lines of my.log.

Wildcard
  • 36,499
Orabîg
  • 237
  • 1
    Please note that this command is watch "($MYCMD | tee -a $MYLOG)", not watch "($MYCMD)" | tee -a $MYLOG. If you got this wrong like I did, the output is going to be very very confusing.

    Another thing to note here is that this command does not add the timestamp of each command's execution by default, so the answer with the loop might still work better for you.

     – ffledgling Aug 19 '19 at 15:34
  • Yes, quotes were meaningful in my answer, as well as they are in computer science ;) – Orabîg Oct 08 '22 at 22:12
5

You could create a script that does it for you. I called mine keep (like, keep doing it) and put it on the bin path.

This is my script:

#!/bin/bash
echo "Repeating command $* every second"
while sleep 1; do
    "$@"
done
Chris Davies
  • 116,213
  • 16
  • 160
  • 287
Alexandre Santos
  • 1,073
  • 2
    You should use "$@" to execute the command (arguments) instead of unquoted $*. This way you'll have the shell keep quoted arguments, etc. as the user would expect them. FTFY. – Chris Davies Mar 03 '16 at 22:56
3

Try the following:

watch -n1 'wc -l my.log >> statistics.log'
HalosGhost
  • 4,790
Andy
  • 31
3

How about

tail -f file.log | pv -rl > /dev/null
Stéphane Chazelas
  • 544,893
1

I came across this question when I was trying to get better/logged output from du -sh $data_path. I used the "while command, do sleep" pattern found here, but used some complex AWK to give the output I wanted.

while du -sh $data_path; do sleep 1; done | awk '
$1 != size {
    size=$1;
    path=$2;
    time=systime();
    seconds=time-prevtime;
    if(seconds < 1000000000){
        seconds=seconds" seconds"
    }else{
        seconds=""
    }
    print size, path, strftime("%m/%d/%Y@%H:%M:%S", time), seconds; 
    prevtime=time
}'

I actually did this as a oneliner, which is why there are semicolons. But to make it readable, I broke it out. The output looks like:

502G /var/lib/cassandra/dump/ 05/22/2018@04:46:17
503G /var/lib/cassandra/dump/ 05/22/2018@04:46:59 42 seconds
504G /var/lib/cassandra/dump/ 05/22/2018@04:47:57 58 seconds
505G /var/lib/cassandra/dump/ 05/22/2018@04:48:55 58 seconds
506G /var/lib/cassandra/dump/ 05/22/2018@04:49:53 58 seconds
507G /var/lib/cassandra/dump/ 05/22/2018@04:50:50 57 seconds
508G /var/lib/cassandra/dump/ 05/22/2018@04:51:46 56 seconds
509G /var/lib/cassandra/dump/ 05/22/2018@04:52:44 58 seconds
510G /var/lib/cassandra/dump/ 05/22/2018@04:53:41 57 seconds
Bruno Bronosky
  • 4,624