Read line from file then delete

Question

How can I read a series of lines from a file and then remove each line?

Obviously I can read like this:

while read -r line; do
echo $line
done < myfile.txt

But if I do this:

while read -r line; do
echo $line
sed '1d' myfile.txt
done < myfile.txt

I only get the very first line

I'm looking for an elegant way to read a line, do something with it and then remove it before moving on to the next line in the file.

Answer 1

I'd prefer to invoke sed just one time then on each line iteration. If you like to change original file you can add -i(--in-place) option to sed.

unset n
while read -r line; do
  echo $line
  : $((n++))
done < myfile.txt
sed "1,$n d" myfile.txt

Answer 2

You can do:

sed -n '$d;w file2' >file1 <<IN
$(cat file1; echo .)
IN

Which will safely write every line from file1 into file2 while simultaneously deleting every line in file1. The shell handles the intermediate temp file securely - it creates one and then deletes it from its filesystem before ever writing a single byte to it.

As soon as sed releases the file-descriptor (at EOF or when it terminates - whichever comes first) the file ceases to exist at all. Unlike the common sed -i option, this does not depend on sed cleaning up after itself (which may not happen if its processing is unexpectedly interrupted) and nor does it carry with it the same permissions inconsistencies the -i option implies.

And the application need not be global as demonstrated - the lines can be either deleted from the original and/or written to the separate output selectively...

sed -n '$d;/pattern1/w file2
           /pattern2/p' >file1 <<IN
$(cat file1; echo .)
IN

...demonstrates how only lines from file1 which match /pattern2/ will be printed back into file1 and only lines from same which match /pattern1/ are written to file2. Any other sed command is of course applicable as well.

It is probably worth noting, though, that for most shells this will not carry over NUL bytes from file1 (which isn't often a concern for sed work) - though such a thing is possible with zsh.

Answer 3

You can replace every line of a file once it was processed correctly with this snippet (do not remove if processing fails, so you can reprocess them if you need it):

while read; do
  if some_command_worked_correctly; then
    sed -i '1,1 d' file-to-read
  fi
done < file-to-read