awk or sed command to match regex at specific line, exit true if success, false otherwise

Question

I need to determine if a file contains a certain regex at a certain line and to return true (exit 0) if found, and otherwise false. Maybe I'm overthinking this, but my attempts proved a tad unwieldy. I have a solution, but I'm looking for maybe others that I hadn't thought of. I could use perl, but I'm hoping to keep this "lightweight" as possible as it runs during a puppet execution cycle.

The problem is common enough: in RHEL6, screen was packaged in a way that limited the terminal width to 80 characters, unless you un-comment the line at 132. This command checks to see if that line has already been fixed:

 awk 'NR==132 && /^#termcapinfo[[:space:]]*xterm Z0=/ {x=1;nextfile} END {exit 1-x}' /etc/screenrc

Note: if the file has fewer that 132 lines, it must exit with false.

I thought sed would be of help here, but apparently then you have to do weird tricks like null-substitutions and branches. Still, I'd like to see a sed solution just to learn. And maybe there is something else I overlooked.

EDIT 1: Added nextfile to my awk solution

EDIT 2: Benchmarks EDIT 3: Different host (idle). EDIT 4: mistakenly used Gile's awk time for optimized-per's run. EDIT 5: new bench

Benchmarks

First, note: wc -l /etc/screenrc is 216. 50k iterations when line not present, measured in wall-time:

• Null-op: 0.545s
• My original awk solution: 58.417
• My edited awk solution (with nextfile): 58.364s
• Giles' awk solution: 57.578s
• Optimized perl solution 90.352s Doh!
• Sed 132{p;q}|grep -q ... solution: 61.259s
• Cuonglm's tail | head | grep -q : 70.418s Ouch!
• Don_chrissti's head -nX |head -n1|grep -q: 116.9s Brrrrp!
• Terdon's double-grep solution: 65.127s
• John1024's sed solution: 45.764s

Thank you John and thank you sed! I am honestly surprised perl was on-par here. Perl loads in a bunch of shared libraries on startup, but as long as the OS is caching them all, it comes down to the parser and byte-coder. In the distant past (perl 5.2?) I found it was slower by 20%. Perl was slower as I originally expected but appeared to be better due to a copy/paste error on my part.

Benchmarks Part 2

The biggest configuration file which has practical value is /etc/services. So I've re-run these benches for this file and where the line to be changed is 2/3rds in the file. Total lines is 1100, so I picked 7220 and modified the regex accordingly (so that in one case it fails, in another it succeeds; for the bench it always fails).

• John's sed solution: 121.4s
• Chrissti's {head;head}|grep solution: 138.341s
• Counglm's tail|head|grep solution: 77.948s
• My awk solution: 175.5s

Answer 1

With GNU sed:

sed -n '132 {/^#termcapinfo[[:space:]]*xterm Z0=/q}; $q1'

How it works

• 132 {/^#termcapinfo[[:space:]]*xterm Z0=/q}

  On line 132, check for the regex ^#termcapinfo[[:space:]]*xterm Z0=. If found quit, q, with the default exit code of 0. The rest of the file is skipped.

• $q1

  If we reach the last line, $, then quit with exit code 1: q1.

Efficiency

Since it is not necessary to read past the 132nd line of the file, this version quits as soon as we reach the 132nd line or the end of the file, whichever occurs first:

sed -n '132 {/^#termcapinfo[[:space:]]*xterm Z0=/q; q1}; $q1'

Handling empty files

The version above will return true for empty files. This is because, if the file empty, no commands are executed and the sed exits with the default exit code of 0. To avoid this:

! sed -n '132 {/^#termcapinfo[[:space:]]*xterm Z0=/q1; q}'

Here, the sed command exits with code 0 unless the the desired string is found in which case it exits with code 1 The preceding ! tells the shell to invert this code to get back to the code we want. The ! modifier is supported by all POSIX shells. This version will work even for empty files. (Hat tip: G-Man)

Answer 2

With POSIX toolchest:

tail -n +132 </etc/screenrc | head -n 1 | grep -q pattern

Answer 3

You can do it in more efficiently in awk: exit as soon as you've hit the relevant line.

awk 'NR==132 {if (/^#termcapinfo[[:space:]]*xterm Z0=/) found=1; exit}
     END {exit !found}' /etc/screenrc

Alternatively, you can use GNU sed (but portable sed doesn't let you specify the exit code).

Alternatively, you can use the Unix philosophy of combining tools together: extract the line you want with head and tail, and pass it to grep.

</etc/screenrc tail -n +132 | head -n 1 |
grep -q '^#termcapinfo[[:space:]]*xterm Z0='

Or you can use sed to extract the desired line:

</etc/screenrc sed -n '32 {p; q;}' |
grep -q '^#termcapinfo[[:space:]]*xterm Z0='

(Both of these rely on the fact that you want the same outcome for an empty line and for a file that's too short.)

For such a small file, the fastest approach is likely to be one that uses a single tool, as the overhead of launching multiple programs will be larger than the performance gain from using special-purpose tools such as head, tail and sed. If you wanted line 132000000, starting off with tail -n +132000000 would likely be faster than anything else.

Answer 4

Some alternatives with ed:

ed -s infile <<\IN
132s/^#termcapinfo[[:space:]]*xterm Z0=/&/
q
IN

or sed+grep:

sed '132!d;q' infile | grep -q '^#termcapinfo[[:space:]]*xterm Z0='

In both cases, if infile has less than 132 lines or if line 132 doesn't match the pattern, the exit code is 1. Both should be quite portable, ed will read the whole file in memory though...

If you're working with huge files then head may be faster then sed e.g.:

{ head -n 131 >/dev/null; head -n 1; } <infile | grep -q '^#termcapinfo[[:space:]]*xterm Z0='

Answer 5

I know you said you didn't want to use perl. I think you're operating under a misconception about how 'lightweight' it is.

You could do this:

#!/usr/bin/env perl

use strict;
use warnings;

open ( my $input_fh, '<', "/etc/screenrc" ) or die $!; 
while ( <$input_fh> ) {
   if ( $. == 132 
   and m/^#termcapinfo[[:space:]]*xterm Z0=/ ) {
       exit 0; 
   }
}

exit 1;

Which you can condense to a one liner:

perl -ne 'exit 0 if $. == 132 and  m/^#termcapinfo[[:space:]]*xterm Z0=/ END { exit 1 }' 

Answer 6

You could always use a couple of greps:

grep -nm 1 "^#termcapinfo[[:space:]]*xterm Z0=" /etc/screenrc | grep -q '^132:'

The -n adds the line number to each matched line in grep's output. For example:

$ seq 11 15 | grep -n 5
5:15

The -m 1 (which, unlike the other two, is not defined by POSIX and might not be available in your grep implementation) makes grep exit after the first match.

So, the first grep looks for lines matching the regex and prints them along with the line number. The second grep will silently (-q) return true if an input line starts with 132:, so it will only be true if the regex matched line 132.

---

Here's another simple Perl approach:

perl -ne '$.==132 && !/^#termcapinfo\s*xterm Z0=/ && exit(1);'

The idea is to exit with a status of 1 only if line 132 doesn't match the regex. It will, therefore, exit with 0 otherwise. You could make it a bit more efficient (but more complex) by only checking the relevant line:

perl -ne '$.==132 && !/^#termcapinfo\s*xterm Z0=/ && exit(1); exit(1) if $.>132'

---

You could also simplify your original awk a little:

awk 'NR==132 && /^#termcapinfo[[:space:]]*xterm Z0=/{exit 0} NR>132{exit 1}'

Answer 7

A workaround with head, tail, wc, and grep. (if loop in bash syntax)

if [[ $(head -n 132 file | wc -l) -eq 123 &&\
    $( head -n 132 file | tail -n 1 |\
    grep '^#termcapinfo[[:space:]]*xterm Z0=') ]] ; then
  echo success
else
  echo fail
fi

Answer 8

For completeness' sake, the ruby solution:

 ruby -e 'while gets do;  if $.==132 ; exit(/^#termcapinfo[[:space:]]*xterm Z0=/?0:1); end; end; exit(1)' /etc/screenrc 

I don't see a way of using ruby -n here without calling at_exit() in every iteration of the file-line read.

The MRI (Matz Ruby Interpreter, 1.8.7) takes insanely long clocking it at 139s.