What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written.
The number of files that are being written is huge, could reach as much as 50,000.
By using inotifywait
, I am monitoring the directory like:
inotifywait -m -e create ~/folder | awk '($2=="CREATE"){print $3}' > ~/output.file
So I am storing names of new files created in ~/output.file
and then using a for loop
for FILE in `cat ~/output.file`
do
cat $FILE >> ~/test.out
done
It works fine, if the rate at which a file is being written (created) in ~/folder
is like 1 file per second.
But the requirement is large, and the rate at which the files are being created is very high, like 500 files per minute (or even more).
I checked the number of files in the ~/folder
after the process is complete, but it does not match the inotifywait
output. There is a difference of like 10–15 files, varies.
Also, the loop
for FILE in `cat ~/output.file`
do
done
doesn't process all the files in ~/output.file
as they are being written.
Can anyone please suggest me an elegant solution to this problem?
inotifywait
runs indefinitely, so I have to kill the process manually. Is there any way of doing this elegantly? The--timeout
option waits only for first event and then exits. Thanks! – rohitkulky Jun 10 '13 at 09:01monitor
andtimeout
together with the current version, you'll have to install thegit
version. Answer updated. – don_crissti Jun 10 '13 at 12:38