6

I have machine on which the files are uploaded by ftp. From this machine I would like to run cronjob and scp/rsync (simply copy them) to different machine in the same network.

The problem is I don't want to copy files which are not coplete (still during transfer) Is there a possibility to check whether a file is complete and only then copy to another server?

countermode
  • 7,533
  • 5
  • 31
  • 58
s1c
  • 597
  • 2
    The usual trick for this is to embed the file's checksum in the filename. – Mark Plotnick Sep 10 '14 at 11:10
  • 2
    Another trick would be to have uploading processes set read permission for the copied file only after the copy is complete. I'm not sure if it is feasible to persuade all upload processes to adhere to this. – Marc van Leeuwen Sep 10 '14 at 11:30

2 Answers2

4

You can use lsyncd:

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync.

You can specify the time out after which a file has changed is to be synced. Set it to e.g. five times the typical upload time and you're probably fine.

Sebastian
  • 8,817
  • 4
  • 40
  • 49
0

If the lsyncd idea isn't to your liking - because "typical upload time" is something bound to break on a medium like the internet, not even considering a wider range of upload sizes (from 32K to 1234T), just think of a resumed ftp transfer. Well, even my idea doesn't help you there, I'm afraid. But how about using lsof?!? 

lsof /home/ftp

Well, after a little perusing in the lsyncd documentation I guess my concerns - born out of the phrasing "typical upload time" - seem unwarranted; the default inotify it watches for is "CloseWrite", which seems fine. YMMV.

Community
  • 1
flowtron
  • 356