How to make a scheduled task based on a file creation time?

Question

The supercomputer I run my Analysis uses SGE to do the job management and requires each job duration not longer than 24 hours. And this feature is causing me quite a headache.

Sadly, I was asked to use checkpoint mechanism to output results and resubmit the script qsub myJobScript. On program's part, it's not difficult for me to have it output a temporary file when the clock time hits 23:30:00 (I give the IO operation half of hour), let's say the output file name is called 1.00137/U and 1.00137/p, etc.
Note: when you submit a job, you never know when it is going to start. but it does have a log file under my folder when it is started.

So my question is how to use crontab or at or bash scripts to submit my job automatically when "new files" are created and detected? Any ideas?

What unix variant is it? For Linux, use an inotify-based mechanism, see Can a bash script be hooked to a file? — Gilles 'SO- stop being evil', Dec 11 '12 at 23:48

score 1 · Answer 1 · edited Apr 13 '17 at 12:36

Run a simple script with cron every 5 minutes. Your crontab could look e.g. like this:

*/5 * * * *  /path/to/your/script

Script pseudocode:

IF (lockfile exists) { exit }
create lockfile
IF (old job has not finished yet) { 
    remove lockfile
    exit
}
IF (new job has already been queued) {
    remove lockfile
    exit
}
schedule new task
remove lockfile

See Correct locking in shell scripts? for how to lock correctly (as race-condition-free as possible).

How to make a scheduled task based on a file creation time?

1 Answers1