How do I print the last sequence of lines between a start and an end pattern?

Question

Answers to this question:

How to grep lines between start and end pattern?

are not concerned with multiple sequences of lines which fall between the match patterns. Thus, for example, sed -n '/startpattern_here/,/endpattern_here/p' will print several sequences of lines which lie between occurrences of these patterns.

However, suppose I want to only print the last such sequences within a file. Can I do this with sed? If not, I guess probably awk? Something else?

Notes:

• You may assume that these sequences are non-overlapping.
• The starting and ending pattern lines should be included in the output.
• Answers making assumptions of lower-complexity patterns are also valid (although not optimal).

Answer 1

This might work, assuming you want a full regular expression test:

awk '/startpattern_here/ {buf="";f=1}
     f{buf=buf $0 "\n"}
     /endpattern_here/ {f=0; lastcomplete=buf}
     END{printf("%s",lastcomplete)}' file.txt

This will ensure that only complete start-stop-patterns will be printed.

Test case:

irrelevant
irrelevant
irrelevant
startpattern_here
relevant_but_dont_show_1
relevant_but_dont_show_1
relevant_but_dont_show_1
endpattern_here
irrelevant
irrelevant
startpattern_here
relevant_but_dont_show_2
relevant_but_dont_show_2
relevant_but_dont_show_2
endpattern_here
irrelevant
irrelevant
startpattern_here
relevant_and_show
relevant_and_show
relevant_and_show
endpattern_here
irrelevant
startpattern_here
incomplete_dont_show

Result:

startpattern_here
relevant_and_show
relevant_and_show
relevant_and_show
endpattern_here

Note If you want to suppress the output of the start and end patterns, just swap the rules /startpattern_here/ { ... } and /endpattern_here/ { ... }, i.e. place the "end pattern" rule first, and the "start pattern" rule just before the END rule.

Answer 2

Combination of tac and awk

tac file \
| awk '
   !p && /endpattern_here/   {p = 1}
    p                        {print}
    p && /startpattern_here/ {exit}
' \
| tac

Answer 3

With Ex (a POSIX editor) that is quite simple:

printf '%s\n' 1 '?END?' '?START?,.p' | ex -s file

• 1 goes to the first line of the file. This is necessary in case END is the last line of the file.

• ?END? seeks backward (wrapping around the end-of-file) for the END, thus finding its last occurrence in the file.

• ?START?,.p prints all from the previous START up to the current address.

Below an example with here-docs instead of printf, just for diversity.

$ cat file
zdk
START
b12
END
kdn
START
000
111
END
START
ddd

$ ex -s file <<EOF
> 1
> ?END?
> ?START?,.p
> EOF
START
000
111
END

Answer 4

It seems I can just use tac:

tac | sed -n '/endpattern_here/,/startpattern_here/ {p; /startpattern_here/q;}' | tac

Thanks goes to @glenn jackman and @Quasimodo for helping me get my sed invocation right.

Answer 5

one way would be to simply store each set, override it with the next one, and print whichever set you have kept once you get to the end:

awk '{ 
        if(/startpattern_here/){
            a=1; 
            lines=$0; 
            next
        } 
        if(a){
            lines=lines"\n"$0
        } 
        if(/end_pattern/){
            a=0
        }
    } 
    END{
        print lines
    }' file

For example, using this test file:

startpattern_here
line 1
line 2
line 3
end_pattern
startpattern_here
line 1b
line 2b
line 3b
end_pattern
startpattern_here
line 1c
line 2c
line 3c
end_pattern

I get:

$ awk '{ if(/startpattern_here/){a=1; lines=$0; next} if(a){lines=lines"\n"$0} if(/end_pattern/){a=0}} END{print lines}' file
startpattern_here
line 1c
line 2c
line 3c
end_pattern

Answer 6

• You can grep out the last range using PCRE flavor of grep in slurp mode.

  grep -zoP '(?ms).*\K^start.*?\nend[^\n]*' file | tr '\0' '\n'
  

• We use the range operator in awk to store and re store once we start a new range. Assuming there is no dangling start pattern line in the vicinity of eof.

  awk '
    /^start/,/^end/ {
      t = (/^start/ ? "" : t ORS) $0
    }
    END { print t }
  ' file
  

• Here we use the tac file to reverse it and then the m?? operator in Perl which matches just once.

  < file tac \
  | perl -lne 'print if m?end? .. m?start?' \
  | tac;
  

• Other alternatives

  < file sed -ne '/start/=;/end/='  \
  | sed -ne 'N;s/\n/,/;$s/$/p/p' \
  | sed -nf - file
  

  < file \
  tac | sed -e '/start/q' |
  tac | sed -e '/end/q'
  

  sed -e '
    /start/,/end/H
    /start/h;g;$q;d
  ' file
  

Answer 7

Most answers here either

1. fail to handle the case where either the start or the end pattern does not exist, or where a line matches both the start and the end pattern.
2. store whole ranges of lines in the memory (unscalable).
3. use some editor like ed or ex which first loads the whole file in the memory.

For the case where the input file is a regular/seekable file (not pipe input), a dumb-simple solution which just gets the last offsets where the start and end patterns matched, and then seeks+reads from there to may be a better idea.

LC_ALL=C awk -v SP=start_pattern -v EP=end_pattern '
   {o+=length+1}
   $0~SP, q=($0~EP) { if(!p) p=o-length; if(q){ l=o+1-(s=p); p=0 } }
   END { if(s && l) system("tail -c +"s" "FILENAME" | head -c "l) }
' file

For the case where the input is from a pipe, you can use a simple pattern range and juggle two temporary files, using close(filename) to rewind them:

... | awk -v SP=start_pattern -v EP=end_pattern -v tmp="$(mktemp)" -v out="$(mktemp)" '
  $0~SP, q=($0~EP){
     print > tmp; if(q){ close(tmp); t=tmp; tmp=out; out=t; }
  }
  END { if(t) system("cat "out); system("rm -f " out " "tmp) }
'

Since any solution will have to parse the whole file before printing anyway (otherwise there's no way to know that it had printed the last range), it makes more sense not to print anything for a file where only the start pattern was found. This is obviously a discutable change from the behaviour of the range operator in sed, awk or perl.

Examples:

seq 1 107 > file
LC_ALL=C awk -v SP=9 -v EP=1 '
   {o+=length+1}
   $0~SP, q=($0~EP) { if(!p) p=o-length; if(q){ l=o+1-(s=p); p=0 } }
   END { if(s && l) system("tail -c +"s" "FILENAME" | head -c "l) }
' file
92
...
100
seq 1 107 | awk -v SP=9 -v EP=1 -v tmp="$(mktemp)" -v out="$(mktemp)" '
  $0~SP, q=($0~EP){
     print > tmp; if(q){ close(tmp); t=tmp; tmp=out; out=t; }
  }
  END { if(t) system("cat "out); system("rm -f " out " "tmp) }
'
92
...
100

Answer 8

$ seq 20 > file
$ awk '/5/{rec=""; f=1} f{rec=rec $0 ORS; if (/8/) f=0} END{if (!f) printf "%s", rec}' file
15
16
17
18

Answer 9

 perl -ne '$x = (/startpattern/../endpattern/ ? $x . $_ : ""); $y=$x if $x and /endpattern/; END { print $y }'

Or, more readably (i.e. not on one line):

#!/usr/bin/perl -n
save a set; could be incomplete
$x = /startpattern/../endpattern/
        ?   $x . $_
        :   ""
    ;
save last complete set seen
if ($x and /endpattern/) {
    $y = $x;
}
print last complete set seen, ignoring any incomplete sets that may have come after
END {
    print $y;
}

Which you run as perl ./script < inputfile

Answer 10

Some possible solutions:

sed: sed -z 's/.*\(StartPattern.*EndPattern[^\n]*\n\).*/\1\n/' file
ed : printf '%s\n' '1;kx' '?^End?;kx' "?^Start?;'xp" | ed -s file
ex : printf '%s\n' '1' '?^End?' "?^Start?,.p" | ex file
awk: awk '/^Start/{s=1;section=""}
s{section=section $0 ORS}
/^End/{complete=section;s=0}
END{printf ("%s",complete)}' file
tac: tac file | sed -n '/^End/,/^Start/{p;/^Start/q}' | tac

---

regex sed

You can match the last occurrence of a pattern between start and end with a regex like:

.*START.*END.*

Then, you can extract the range including the delimiters with a parentheses.

.*\(START.*END\).*

That will work in sed (as it may use the replace s///) but require GNU sed to make the whole file one string (using the -z option):

sed -z 's/.*\(StartPattern.*EndPattern[^\n]*\n\).*/\1\n/' file    

ed

It is possible to search backwards in ed with ?regex?. So, we can search backwards for EndPattern (to ensure the pattern is complete and we are at the last one) and then search also backward to the previous StartPattern.

printf '%s\n' '?^End?;kx' '?^Start?;kx' '.;/End/p' | ed -s file

The ;kx is used to avoid that ed prints the selected line.

That would fail if the last line is End, to avoid that, start on the first line and search backward for End.

And, since the limits are being marked, we can use a simpler range:

printf '%s\n' '1;ky' '?^End?;ky' '?^Start?;kx' "'x;'yp" | ed -s file

Or,

printf '%s\n' '1;kx' '?^End?;kx' "?^Start?;'xp" | ed -s file

That is assuming that at least one complete section of Start -- End exists. If there is none, the script will fail.

I have seen several uses of ?Start?,?End?. That may fail in several ways because it doesn't mean "find the next End after what was found by Start. Compare:

$ printf '%s\n' 1 '?START?,?END?p' | ex -s <(printf '%s\n' 111 START 222 END 333 END 444)
START
222
END
333
END
$ printf '%s\n' 1 '?START?,/END/p' | ex -s <(printf '%s\n' 111 START 222 END 333 END 444)
START
222
END
ex
The command from ed could be simplified to work in ex:
printf '%s\n' '1' '?^End?' '?^Start?,.p' | ex file
awk
We can store each complete section Start to End in one variable and print it at the end.

awk ' /^Start/{s=1;section=""} # If there is an start, mark a section. s{section=section $0 ORS} # if inside a section, capture all lines. /^End/{complete=section;s=0} # If a section ends, unmark it but store. END{printf ("%s",complete)}' file # Print a complete section (if one existed).

# tac
We can reverse the whole file (line by line) and then print only the **first** section that starts at `End` and ends at `Start`. Then reverse again:
tac file | sed -n '/^End/,/^Start/{p;/^Start/q}' | tac
The /^Start/q exists sed to ensure that only the first section is printed.
Note that this will print everything from the last End to the start of the file if there is no Start to be found (instead of just not printing).
test file
Tested with (at least) this file (and others):

$ cat file3 Don't print 1 Don't print 2 Don't print 3 StartPattern_here-1 Inside Pattern but Don't print 1-1 Inside Pattern but Don't print 1-2 Inside Pattern but Don't print 1-3 EndPattern_here-1

Lines between 1 and 2 - 1 Lines between 1 and 2 - 2 Lines between 1 and 2 - 3

StartPattern_here-2 Inside Pattern but Don't print 2-1 Inside Pattern but Don't print 2-2 Inside Pattern but Don't print 2-3 EndPattern_here-2

Lines between 2 and 3 - 1 Lines between 2 and 3 - 2 Lines between 2 and 3 - 3

StartPattern_here-3 Inside Pattern, Please Do print 3-1 Inside Pattern, Please Do print 3-2 Inside Pattern, Please Do print 3-3 EndPattern_here-3

Lines between 3 and 4 - 1 Lines between 3 and 4 - 2 Lines between 3 and 4 - 3

StartPattern_here-4 This section has an start but not an end, thus, incomplete. Lines between 4 and $ - 1 Lines between 4 and $ - 2 Lines between 4 and $ - 3

Answer 11

Here is a solution trying to handle all cases, including no printing for no block found, and be efficient in memory and execution time. There is no writing line by line in this solution, no processing of every line and no lines buffering.

#!/bin/bash
sp="startpattern_here"
ep="endpattern_here"
f="file"
range=$(tac "$f" | grep -n "$sp|$ep" | awk -F: -v sp="$sp" -v ep="$ep"

        '$2 ~ sp && prev ~ ep {s=$1; print s,e; exit} {prev=$2; e=$1}')
if [[ "$range" ]]; then
    # echo "Counting from the end => start: ${range% } end: ${range# }"
    tail -n "${range% }" "$f" | head -n "${range# }"
else
    echo "No blocks found" 1>&2
fi

Explanation and example:

> cat file
startpattern_here
text
endpattern_here
startpattern_here
text
startpattern_here
42
endpattern_here
text
endpattern_here

In the worst case scenario, we have to search the whole file for a complete answer, so we use the fast grep for that. We start searching from the end, so it will get something like this:

1:endpattern_here
3:endpattern_here
5:startpattern_here
7:startpattern_here
8:endpattern_here
10:startpattern_here

which is piped to awk to decide if there is a valid last block or not. Note that here awk is being used for simple programming, not for the actual text processing. For a large input, grep is faster than searching the file with awk or even more, writing line by line with awk or sed.

Also, in case a block between patterns is detected quickly close at the end, awk is exiting and closing its pipe, so the previous sequence is also exiting, without searching the whole file.

This way, we get the range, counting from the end, and finally tail and head seek() to those line numbers and "cat" the content. In case of empty range, there is no standard output.

startpattern_here
42
endpattern_here

Answer 12

Fast and simple sed-only solution. Most other solutions are either wasting resources by double-tac-ing, or even worse, loading whole input into memory at once, or doing multiple pass processing in some way.

This processes text line-by-line, so we only requires memory for one copy of matched block, and we don't fork and exec other stuff which would do even more extra processing. As a bonus, it is quite readable and understandable (well, as far as any sed script can be).

Instead of your: sed -n '/startpattern_here/,/endpattern_here/p' you do this:

sed -n '/startpattern_here/,/endpattern_here/H; /startpattern_here/h; ${g;p}'

Explanation (note: anything after ; is independent of previous commands, unless grouped with { and }):

• first part /startpattern_here/,/endpattern_here/H is mostly similar to one in your question, but instead of outright printing to stdout everything found between start and end patterns, it instead appends that text to "hold space" (H).

• /startpattern_here/h notices when NEW match starts, and erases previous hold space by overwriting it (h) with current pattern space. Note that next line in file will of course start executing all of our commands from scratch, which will keep appending to hold space (see above point) - result being that we will always keep in hold space only last matched block.

• ${g;p} - $ address matches only on last line in file, so anything between { and } is executed only when we're finished with processing file. Here we simply print content of hold space (by g - copying hold space to pattern space, and p - printing pattern space)

for example, to get last Debian package basic info:

% sed -n '/^Package/,/^Section/H; /^Package/h; ${g;p}' /var/lib/dpkg/status
Package: zsh-common
Status: install ok installed
Priority: optional
Section: shells