How can I copy lines from a file every 60 seconds, and continue from last copied

Question

I need to copy x lines from a file every 60 seconds, and after 60 sec, continue to copy for another 60 sec, restarting from last copied?

#!/bin/bash
while true; do
  sed -n -e '1,10000' input.yml > output.txt 
  sleep 60 
done

thanks

Answer 1

If you want to overwrite output.txt in each run:

#! /bin/sh -
while true; do
  head -n 10000 > output.txt
  sleep 60
done < input.yml

If not:

#! /bin/sh -
while true; do
  head -n 10000
  sleep 60
done < input.yml > output.txt

Those copy up to 10000 lines every 60 seconds. If there aren't that many lines, it will not wait until there are 10000.

They assume a head implementation that leaves the cursor within the input just after the list byte they output. That's a POSIX requirement if the input file is seekable (like when input.yml is a regular file).

If you wanted to wait for those 10000 lines at each iteration, you could do it (on a GNU system) with:

tail -fn+1 input.yml |
  while true; do
    sed -u 10000q > output.txt
    sleep 60
  done

Here, the input is a pipe which is not seekable. So we need a head implementation that can read one byte at a time until the 10000^th newline character. Hence GNU sed with its -u that does just that.

Another approach would be:

tail -fn+1 input.yml | awk -v out=output.txt -v n=10000 '
  {print > out}
  NR % n == 0 {close(out); system("sleep 60")}'

(if your awk is mawk, you need to pass -W interactive). That one doesn't need to read one byte at a time because it's always the same invocation of the same command that reads the input. Note that awk executes a shell each time to run that sleep 60 command.