How to cut a file starting from the line in which a certain pattern occurs?

Question

I have a gazillion files which need to be reduced in size. I found that most (not all) files have an end section which can be cut without losing information:

Data 1
Data 2
something_unimportant_here END DATA
Rubbish 1
Rubbish 2

How can I edit a file (end hence, all) by deleting the line including "END DATA" and all following, in-place, changing only those the files that contain the pattern, thereby minimising write access to the disk (many, many files and slow disk).

If possible, I would like to add a new last line to the file (my own end tag) so that the file's syntax stays correct -- again, only in those files containing the pattern.

I was thinking of using ed, like

echo ',s/END DATA/ ???? '\\n'q'\\n'wq' | ed "$file"

but cannot seem to manage the ???? part correcty.

Expected output:

Data 1
Data 2
NEW END

Answer 1

You should be able to do it by just truncating the file in place without having to write a new copy of the file like sed -i/perl -i/ed/gawk -i /usr/share/awk/inplace.awk¹ would do. With perl:

find . -name '*.txt' -type f -exec perl -ne '
  BEGIN{@ARGV=map{"+<$_"}@ARGV} # open files in read+write mode in the
                                # while(<>) loop implied by -n
  if (/END DATA/) {
    seek ARGV,-length,1; # back to beginning of matching line
    print ARGV "NEW END\n";
    truncate ARGV, tell ARGV;
    close ARGV; # skip to next file
  }' {} +

That minimises the I/O as perl stops reading as soon as it finds a match, and NEW END\n is the only thing it writes. It also writes in place, so the files metadata (ownership, permission, acls, sparseness...) are preserved and hard links are not broken.

With -exec {} + we also minimise the number of perl invocations.

---

^{¹ do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'}

Answer 2

It sounds like the sequence of commands you're looking for is

/END DATA/,$d
q
.a
NEW END
.
wq

or as a one-liner

printf '%s\n' '/END DATA/,$d' 'q' '.a' 'NEW END' '.' 'wq'

(You can replace wq with ,p for testing.)

Ex. given

$ cat file
Data 1
Data 2
something_unimportant_here END DATA
Rubbish 1
Rubbish 2

then

$ printf '%s\n' '/END DATA/,$d' 'q' '.a' 'NEW END' '.' 'wq' | ed -s file

gives

$ cat file
Data 1
Data 2
NEW END

Answer 3

With GNU grep and GNU sed

grep -lZ 'END DATA' *.txt | xargs -0 sed -i -e '/END DATA/,${//i foo' -e 'd}'

where *.txt assumes all your files are in current directory ending with .txt extension. If you need to recursively search for files, GNU grep also supports -r/-R options.

/END DATA/,$ range of lines to operate

//i foo here // will match the previously used regex, i.e. /END DATA/ and i command will add the new ending marker as needed

as i command has to be separated by newline, -e option is used to separate the d command to delete all lines matched by the range

as an alternate, you can also use this, but only one file will be passed at a time to sed:

grep -lZ 'END DATA' *.txt | xargs -0 -n1 sed -i -e '/END DATA/{i foo' -e 'Q}'

Answer 4

Keep it simple and just use awk for the file manipulation part, e.g., with GNU find, awk, grep, and xargs:

find . -type f -exec grep -lZ 'END DATA' {} + |
    xargs -r0 awk -i inplace '/END DATA/{nextfile} 1'

or to print your own end tag at the end of every file:

... |
  xargs -r0 awk -i inplace '/END DATA/{print "NEW END"; nextfile} 1'

Answer 5

This python 3.8 solution is loosely based on Stephane's in-place truncate solution with a couple of differences 1. The code doesn't rely on external utilities for directory traversal 2. Files are memory-mapped for ease of locating the END DATA string

Place the code in a .py file and pass the directory name as a parameter

import mmap
import os
import sys
from contextlib import closing

def yield_all_files(dir_):
    for root, dir_, files in os.walk(dir_):
        yield from (os.path.join(root, file_) for file_ in files if file_.endswith('.txt'))

if __name__ == '__main__':
    old = b'END DATA'
    new = b'NEW END\n'
    dir_ = sys.argv[1]
    for file_ in yield_all_files(dir_):
        with open(file_, mode='r+b') as f:
            with closing(mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_WRITE)) as mm:
                if (loc := mm.find(old)) > -1:
                    mm.seek(loc)
                    mm.write(new)
                    mm.resize(mm.tell()) 

Answer 6

Combining Sundeep’s answer and Ed Morton’s answer, but without xargs:

find . -type f -name '*.txt' …(other criteria)… -exec grep -q 'END DATA' {} ';'  -a  \
                                -exec sed -i -e '/END DATA/,${//i NEW END' -e 'd}' {} +

• find, of course, selects files.  By default, it searches the specified director(ies) recursively.  To search only the current directory, add -maxdepth 1 after the ..
• grep -q quietly exits with a “success” exit status if a file contains the pattern being searched for (END DATA) and “false” otherwise.
• -a means “AND”, like && on a shell command line.  It means “do the following thing if (only if) the previous thing succeeded.”  Actually, it’s the default conjunction operator between find predicates (tests/actions), so you can leave it out.  I included it only for clarity.
• So the sed command, which is copied verbatim from Sundeep’s answer (but with foo changed to NEW END), is executed only on files that contain the END DATA string and satisfy the other find tests.
• -exec … + causes sed to be invoked once with multiple files, as xargs does it.
  Note that we can’t use -exec … + with the grep command because it doesn’t allow testing exit status.

Answer 7

Using awk to find the offset of the pattern, and dd to truncate the file at that point and append the new trailer:

# usage truncatoo pattern new_end find_args ...
truncatoo(){
    pat=$1; shift; tail=$1; shift
    LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{
           gsub(q,q"\\"q q,FILENAME);
           system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null");
           exit
       }
       {l+=length()+1}
    ' {} \;
}

truncatoo '/END DATA/' 'NEW END\n' file.txt
truncatoo '/END DATA/' 'NEW END\n' . -type f -name '*.txt'

With an awk implementation which supports nextfile (gawk, bwk, some versions of mawk [1]), that could be done more efficiently by passing batches of files to awk:

# usage truncatoo pattern new_end find_args ...
truncatoo(){
    pat=$1; shift; tail=$1; shift
    LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{
           gsub(q,q"\\"q q,FILENAME);
           system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null");
           l=0; nextfile
       }
       {l+=length()+1}
    ' {} +
}

$ file="a'b\$q  * r"; seq 1 100 >"$file"
$ truncatoo /7/ 'CUT\n' "$file"; cat "$file"
1
2
3
4
5
6
CUT

Instead of the icky 2>/dev/null status=noxfer could be used with dd implementations which support it.

The quoting kludge and environment variables passing is a mess, it could use some improvement.

[1]: according to the GNU awk manual, it should be supported in mawk too. However, the older version of mawk from Debian 10 does not support it.

Answer 8

If this is a one time task, it is convenient to use vi for this:

First, locate the line with the needed content (by using search / or ?)

To delete all lines following the current line to the end of file, press d G.

To delete all lines from the start of the file to the current line, press d gg.

Save and quit :wq