Run unix command precisely at very short intervals WITHOUT accumulating time lag over time

Question

Question

I'd like to be able to run a UNIX command precisely every second over a long time period.

I need a solution, which does not lag behind after a certain time, because of the time the command itself needs for execution. sleep, watch, and a certain python script all failed me in this regard.

On the microcontroller's such as the http://Arduino.cc I'd do that through hardware clock interrupts. I'd like to know whether there is a similar time-precise shell script solution. All the solutions which I found within StackExchange.com, resulted in a noticeable time lag, if run over hours. See details below.

Practical purpose / application

I want to test whether my network connection is continuously up by sending timestamps via nc (netcat) every 1 second.

Sender:

precise-timestamp-generator | tee netcat-sender.txt | nc $receiver $port

Receiver:

nc -l -p $port > netcat-receiver.txt

After completion, compare the two logs:

diff netcat-sender.txt netcat-receiver.txt

The diffs would be the untransmitted timestamps. From this I would know at what time my LAN / WAN / ISP makes troubles.

---

Solution SLEEP

while [ true ]; do date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done | tee timelog-sleep.txt

Gets a certain offset over time, as the command within the loop also takes a little time.

Precision

cat timelog-sleep.txt

2012-07-16 00:45:16
[...]
2012-07-16 10:20:36

Seconds elapsed: 34520

wc -l timelog-sleep.txt

Lines in file: 34243

Precision summarized:

• 34520-34243 = 277 timing problems
• 34520/34243 = 1.008 = 0.8 % off

Solution REPEAT PYTHON

Found at: Repeat a Unix command every x seconds forever

repeat.py 1 "date '+%Y-%m-%d %H:%M:%S'" >> timelog-repeat-py.txt

Supposed to avoid the time offset, but fails to do so.

Precision

wc -l timelog-repeat-py.txt

2012-07-16 13:42:44
[...]
2012-07-16 16:45:24

Seconds elapsed: 10960

wc -l timelog-repeat-py.txt

Lines in file: 10859

Precision summarized:

• 10960-10859 = 101 timing problems
• 10960/10859 = 1.009 = 0.9 % off

Solution WATCH

watch -n 1 "date '+%Y-%m-%d %H:%M:%S' >> ~/Desktop/timelog-watch.txt"

Precision

wc -l timelog-watch.txt
2012-07-16 11:04:08
[...]
2012-07-16 13:25:47

Seconds elapsed: 8499

wc -l timelog-watch.txt

Lines in file: 8366

Precision summarized:

• 8499-8366 = 133 timing problems.
• 8499/8366 = 1.016 = 1.6 % off.

Answer 1

Have you tried watch with the parameter --precise?

watch -n 1 --precise "date '+%Y-%m-%d %H:%M:%S.%N' >> ~/Desktop/timelog-watch.txt"

From the man page:

Normally, this interval is interpreted as the amout of time between the completion of one run of command and the beginning of the next run. However, with the -p or --precise option, you can make watch attempt to run command every interval seconds. Try it with ntptime and notice how the fractional seconds stays (nearly) the same, as opposed to normal mode where they continuously increase.

The parameter might not be available on your system, though.

You should also consider what should happen when the execution of your program needs more than one second. Should the next scheduled execution be skipped, or should it be run late?

Update: I ran the script for some time, and it didn't loose a single step:

2561 lines
start: 2012-07-17 09:46:34.938805108
end:   2012-07-17 10:29:14.938547796

Update: The --precise flag is a Debian addition, the patch is however rather simple: http://patch-tracker.debian.org/patch/series/view/procps/1:3.2.8-9squeeze1/watch_precision_time.patch

Answer 2

The POSIX ualarm() function lets you schedule the kernel to periodically signal your process, with microsecond precision.

Whip up a simple program:

 #include<unistd.h>
 #include<signal.h>
 void tick(int sig){
     write(1, "\n", 1);
 }
 int main(){
     signal(SIGALRM, tick);
     ualarm(1000000, 1000000); //alarm in a second, and every second after that.
     for(;;)
         pause();
 }

Compile

 gcc -O2 tick.c -o tick

Then attach it to whatever you need done periodically like so:

./tick | while read x; do
    date "+%Y-%m-%d %H:%M:%S"
done | tee timelog-sleep.txt

Answer 3

crontab has a resolution of 1 minute. If you're fine with lag time accumulating per that minute and then resetting on the next minute, this basic idea could work:

* * * * * for second in $(seq 0 59); do /path/to/script.sh & sleep 1s;done

Note that script.sh is also run in the background. This should help to minimize the lag that accumulates with each iteration of the loop.

Depending on how much lag sleep generates, there is however the chance of second 59 overlapping with second 0 of the next minute.

EDIT to toss in some results, in the same format as in the question:

$ cat timelog-cron
2012-07-16 20:51:01
...
2012-07-16 22:43:00

1 hour 52 minutes = 6720 seconds

$ wc -l timelog-cron
6720 timelog-cron

0 timing problems, 0% off. Any time accumulation resets every minute.

Answer 4

Your problem is that you're sleeping for a fixed amount of time after you run your program without taking in to consideration the amount of time that has elapsed since the last time you slept.

You can do this is bash or any other programming language, but the key is to use the clock to determine how long to schedule the next sleep. Before you sleep, check the clock, see how much time you have left, and sleep the difference.

Because of process scheduling compromises, you are not guaranteed to wake up right on the clock tick, but you should be fairly close (within a few ms unloaded, or within a few hundred ms under load). And you won't accumulate error over time because each time you are re-synchronizing on every sleep cycle and removing any accumulated error.

If you need to hit the clock tick exactly, then what you're looking for is a real time operating system, which are designed exactly for this purpose.

Answer 5

How does this Perl script I just whipped up work?

#!/usr/bin/perl

use strict;
use warnings;
use Time::HiRes qw/time sleep/;

sub launch {
    return if fork;
    exec @_;
    die "Couldn't exec";
}

$SIG{CHLD} = 'IGNORE';

my $interval = shift;
my $start = time();
while (1) {
    launch(@ARGV);
    $start += $interval;
    sleep $start - time();
}

Use: perl timer.pl 1 date '+%Y-%m-%d %H:%M:%S'

It has been running 45 minutes without a single skip, and I suspect it will continue to do so unless a) system load becomes so high that fork() takes more than a second or b) a leap second is inserted.

It cannot guarantee, however, that the command runs at exact second intervals, as there is some overhead, but I doubt it is much worse than an interrupt-based solution.

I ran it for about an hour with date +%N (nanoseconds, GNU extension) and ran some statistics on it. The most lag it had was 1 155 microseconds. Average (arithmetic mean) 216 µs, median 219 µs, standard deviation 42 µs. It ran faster than 270 µs 95% of the time. I don't think you can beat it except by a C program.

Answer 6

I've always just given up on having something run exactly on interval. I think you'll have to write a C program, and pay very careful attention to not exceeding the portion of the 1-second interval with your own code. You'll probably have to use threading or multiple, inter-communicating processes to get this to work. Take care to avoid thread-starting or process-starting time overhead.

One reference that seems relevant dates to 1993: A Randomized Sampling Clock for CPU Utilization Estimation and Code Profiling You'll want to take a look at the appendix "Adversary Source Code" to see how they accurately measured time intervals, and "woke up" their program at just the correct time. Since the code is 19 years old, it probably won't port directly or easily, but if you read it and try to understand it, the principles involved might guide your code.

EDIT: Found another reference that might help: Effects of Clock Resolution on the Scheduling of Interactive and Soft Real-Time Processes That should help you with any theoretical background.

Answer 7

Take a look at nanosleep() (from http://linux.about.com/library/cmd/blcmdl2_nanosleep.htm). Instead of making your program sleep 1 second, make it sleep (1 - ammount it takes to run) seconds. You'll get a much better resolution.

Answer 8

Try running your command in the background so it does not affect the loop timing so much, but even that will not be enough if you don't want any accumulation for long periods of time as there is surely a few millisecond cost associated with it.

So, this is likely better, but also likely still not good enough:

while [ true ]; do date "+%Y-%m-%d %H:%M:%S" & sleep 1; done | 
tee timelog-sleep.txt

On my computer this gave 2 errors in 20 minutes or 0,1 per minute, which is roughly a low five fold improvement over your run.

Answer 9

With ksh93 (which has a floating point $SECONDS and a builtin sleep)

typeset -F SECONDS=0
typeset -i i=0
while true; do
   cmd
   sleep "$((++i - SECONDS))"
done

The same script will work with zsh as well but will invoke your system's sleep command. zsh has a zselect builtin, but with 1/100 resolution only.

Answer 10

Ugly but it works. You should probably rethink the design of your program if you need a loop like this. It basically checks if the current whole second is equal to the previous checked one and prints the number of nanoseconds since change of the second. The accuracy is influenced by the sleep .001.

while true; do T=$( date +%s ); while [[ $T -eq $( date +%s ) ]]; do sleep .001; done; date "+%N nanoseconds late"; done

Accuracy is in the milliseconds, provided that the 'payload' date "+%N nanoseconds late" does not take longer than just under a second. You can lower CPU load by increasing the sleep period or if you really don't mind just replace the sleep command by true.

002112890 nanoseconds late
001847692 nanoseconds late
002273652 nanoseconds late
001317015 nanoseconds late
001650504 nanoseconds late
002180949 nanoseconds late
002338716 nanoseconds late
002064578 nanoseconds late
002160883 nanoseconds late

This is bad practice because you basically make thee CPU poll for an event and you are wasting CPU cycles. You probably want to attach to a timer interrupt (not possible from bash) or use dedicated hardware like a microcontroller. A PC and its operating system are not designed for high timing accuracy.

Answer 11

Another method would be to use a suspend in a loop and send SIGCONT from a precise external program. Sending a signal is very lightweight and will have much less latency than executing something. You could also pre-queue a bunch of commands with the "at" command, hardly anybody uses "at" anymore I'm not sure how precise it is.

If precision is critical and you want to get serious about this, this sounds like the sort of application where typically you would use RTOS, which could be done under Linux with RT-Preempt patched kernel, that will give you the precision and some measure of interrupt control, but it may be more bother than it's worth.

https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO

Xenomai also might be helpful, it's a full RTOS implementation and is ported for x86 and x86_64, but there's some programming involved.

http://www.xenomai.org/index.php/Main_Page

Answer 12

Here's one that's a bash script and highly accurate. It uses usleep for microsecond precision.

http://wiki.junkemailfilter.com/index.php/How_to_run_a_Linux_script_every_few_seconds_under_cron

There's 3 scrips there. Look at the one at the bottom. Will do up to about 2400 executions per minute with reasonable accuracy. And it's really simple.

Answer 13

This one can run at least 100 times a second with very accurate resolution.

The existence of the directory of the number of loops per minute creates the schedule. This version supports microsecond resolution assuming your computer can handle it. The number of executions per minute does not have to to evenly divisible by 60 nor is it limited to 60 I have tested it to 6000 and it works.

This version can be installed in the /etc/init.d directory and run as a service.

#! /bin/sh

# chkconfig: 2345 91 61
# description: This program is used to run all programs in a directory in parallel every X times per minute. \
#              Think of this program as cron with microseconds resolution.

# Microsecond Cron
# Usage: cron-ms start
# Copyright 2014 by Marc Perkel
# docs at http://wiki.junkemailfilter.com/index.php/How_to_run_a_Linux_script_every_few_seconds_under_cron"
# Free to use with attribution

# The scheduling is done by creating directories with the number of"
# executions per minute as part of the directory name."

# Examples:
#   /etc/cron-ms/7      # Executes everything in that directory  7 times a minute
#   /etc/cron-ms/30     # Executes everything in that directory 30 times a minute
#   /etc/cron-ms/600    # Executes everything in that directory 10 times a second
#   /etc/cron-ms/2400   # Executes everything in that directory 40 times a second

basedir=/etc/cron-ms

case "$1" in

   start|restart|reload)
   $0 stop
   mkdir -p /var/run/cron-ms
   for dir in $basedir/* ; do
      $0 ${dir##*/} &
   done
   exit
   ;;

   stop)
   rm -Rf /var/run/cron-ms
   exit
   ;;

esac

# Loops per minute is passed on the command line

loops=$1
interval=$((60000000/$loops))

# Just a heartbeat signal that can be used with monit to verify it's alive

touch /var/run/cron-ms

# After a restart the PIDs will be different allowing old processes to terminate

touch /var/run/cron-ms/$$

# Sleeps until a specific part of a minute with microsecond resolution. 60000000 is full minute

usleep $(( $interval - 10#$(date +%S%N) / 1000 % $interval ))

# Deleting the PID files exit the program

if [ ! -f /var/run/cron-ms/$$ ]
then
   exit
fi

# Run all the programs in the directory in parallel

for program in $basedir/$loops/* ; do
   if [ -x $program ] 
   then
      $program &> /dev/null &
   fi
done

exec $0 $loops

Answer 14

I'd go with a small C program:

#include <sys/time.h>
#include <unistd.h>

int main(int argc, char **argv, char **envp)
{
    struct timeval start;
    int rc = gettimeofday(&start, NULL);
    if(rc != 0)
            return 1;

    for(;;)
    {
        struct timeval now;
        rc = gettimeofday(&now, NULL);
        useconds_t delay;
        if(now.tv_usec < start.tv_usec)
            delay = start.tv_usec - now.tv_usec;
        else
            delay = 1000000 - now.tv_usec + start.tv_usec;
        usleep(delay);
        pid_t pid = fork();
        if(pid == -1)
            return 1;
        if(pid == 0)
            _exit(execve(argv[1], &argv[1], envp));
    }
}

This program expects the program to call with full path as its first argument, and passes on any remaining arguments. It will not wait for the command to finish, so it will happily start multiple instances.

Also, the coding style here is really sloppy, and a number of assumptions are made that may or may not be guaranteed by applicable standards, i.e. the quality of this code is "works for me".

This program will get somewhat longer or shorter intervals when the clock is adjusted by NTP or by manually setting it. If the program should handle this, POSIX provides timer_create(CLOCK_MONOTONIC, ...) which is unaffected by this.

Answer 15

You should be keeping track of the current time and be comparing it to the start time. So you sleep a calculated ammount of time each iteration, not a fixed amount. In this way you wont accumulate timing errors and shift away from where you should be because you reset your timings each loop to absolute time from the start.

Also some sleep functions return early if there is an interupt, so in this case you will have to call your sleep method again until the full amount of time has passed.