enumerate all matches in file with sed or perl

Question

I have next file:

:~$ cat test
        <Set Id="16">
                <Ver="44" Sniff="no" B2vpnId="" Collision="no">
        </Set>        
        <Set Id="17">
                <Ver="22" Sniff="no" B2vpnId="" Collision="no">
        </Set>

and I need to increment B2vpnId value in file, I found a way how to get this change in output:

:~$ i=1;while read -r line; do echo "$line"|sed -e 's/B2vpnId=""/B2vpnId="'"vpn$i"'"/';((i+=1));done <<< "$(cat test|grep B2vpnId=)"
<Ver="44" Sniff="no" B2vpnId="vpn1" Collision="no">
<Ver="22" Sniff="no" B2vpnId="vpn2" Collision="no">

but since I have many other config lines I need this change in file itself and cannot find a way to do it.

Welcome, this is better suited for awk, is there a reason why you would not use it? — schrodingerscatcuriosity, Nov 30 '21 at 10:14
Don't use a shell script for this. If you must use a shell script, use it as a wrapper around awk or perl or something - don't use a while read loop to pipe individual lines into sed, awk, grep, perl, they're all far more capable of processing multi-line input than bash is. See Why is using a shell loop to process text considered bad practice? — cas, Nov 30 '21 at 10:28
If the input is XML, could you please update the document in the question so that it is a representative sample from your actual file? The current document is not well-formed XML (missing root node, invalid Ver tags, and missing closing tags). This operation is trivial with an XML parser, but not if the document is faulty. — Kusalananda, Nov 30 '21 at 10:33

cas · Answer 1 · 2021-11-30T14:08:04.313

Using perl's /e regex modifier to evaluate the replacement portion of the s/search/replace/ operator as perl code:

$ perl -p -e 's/B2vpnId=""/sprintf "B2vpnId=\"vpn%i\"", ++$i/e' test 
        <Set Id="16">
                <Ver="44" Sniff="no" B2vpnId="vpn1" Collision="no">
        </Set>
        <Set Id="17">
                <Ver="22" Sniff="no" B2vpnId="vpn2" Collision="no">
        </Set>

If you want it to change the files on disk (rather than just output to stdout), use perl's -i option.

If you are processing multiple files and want it to reset the counter ($i) after each file, add ; $i=0 if eof to the end of the script:

$ cp test test2
$ perl -p -e 's/B2vpnId=""/sprintf "B2vpnId=\"vpn%i\"", ++$i/e; $i=0 if eof' test test2
        <Set Id="16">
                <Ver="44" Sniff="no" B2vpnId="vpn1" Collision="no">
        </Set>
        <Set Id="17">
                <Ver="22" Sniff="no" B2vpnId="vpn2" Collision="no">
        </Set>
    &lt;Set Id=&quot;16&quot;&gt;
            &lt;Ver=&quot;44&quot; Sniff=&quot;no&quot; B2vpnId=&quot;vpn1&quot; Collision=&quot;no&quot;&gt;
    &lt;/Set&gt;
    &lt;Set Id=&quot;17&quot;&gt;
            &lt;Ver=&quot;22&quot; Sniff=&quot;no&quot; B2vpnId=&quot;vpn2&quot; Collision=&quot;no&quot;&gt;
    &lt;/Set&gt;

Without the $i=0 if eof statement, the second copy of test would have B2vpnIDs of "vpn3" and "vpn4".

BTW, if you want the counter to start from a different number, e.g. 10, then add BEGIN { $i=9 }; to the beginning of the script. BEGIN blocks are executed only once before any of the input is read. The rest of the script is executed repeatedly, once for each input line. BTW, there are other kinds of code blocks that are only executed once - from man perlsyn:

When a block is preceded by a compilation phase keyword such as BEGIN, END, INIT, CHECK, or UNITCHECK, then the block will run only during the corresponding phase of execution. See perlmod for more details.

Remember that $i will be incremented before it is used each time (because the script is using ++$i rather than $i++, which would increment it after each use), so subtract 1 from the starting number.

Also remember to change the $i=0 if eof to match the BEGIN block, e.g., $i=9 if eof if you're using that.

$ perl -p -e 'BEGIN { $i=9 };
              s/B2vpnId=""/sprintf "B2vpnId=\"vpn%i\"", ++$i/e;
              $i=9 if eof' test

Or, since we're now using a BEGIN block to initialise $i, we can post-increment it (and use another variable for the starting value so we only have to change it in one place).

$ perl -p -e 'BEGIN { $start = 10; $i = $start };
              s/B2vpnId=""/sprintf "B2vpnId=\"vpn%i\"", $i++/e;
              $i = $start if eof' test

or even set it to a default value of 1, then if the first arg isn't an existing file set it to the first arg :

$ perl -p -e 'BEGIN {
                $start = 1;
                $start = shift unless -f $ARGV[0];
                $i = $start
              };
          s/B2vpnId=&quot;&quot;/sprintf &quot;B2vpnId=\&quot;vpn%i\&quot;&quot;, $i++/e;

          $i = $start if eof' 10 test*

schrodingerscatcuriosity · Answer 2 · 2021-11-30T10:52:45.760

1

This is better suited for awk:

$ awk 'BEGIN { c=1 } /B2vpnId/ { sub(/vpnId="/, "vpnId=\"vpn"c,$0); c++ }1' test
        <Set Id="16">
                <Ver="44" Sniff="no" B2vpnId="vpn1" Collision="no">
        </Set>        
        <Set Id="17">
                <Ver="22" Sniff="no" B2vpnId="vpn2" Collision="no">
        </Set>

To actually change the file (GNU awk):

$ awk -i inplace <rest of command>

Note: See Useless use of cat;

edited Nov 30 '21 at 10:52

answered Nov 30 '21 at 10:25

schrodingerscatcuriosity

12,396

Thx, very useful and simple, awk is perfect tool for my case - it is easy to understand its syntax. – denys.by Nov 30 '21 at 10:47

score 0 · Answer 3 · answered Nov 30 '21 at 10:26

The simplest way would be to re-direct your output to a temp-file and then move the temp-file over the original file.

I am, however a bit surprised by your complicated command line.

i=1
while read -r line; do
    echo "$line"|sed -e 's/B2vpnId=""/B2vpnId="'"vpn$i"'"/'
    ((i+=1))
done <<< "$(cat test|grep B2vpnId=)"

A simpler way would be:

i=1
grep 'B2vpnId=' test |
while read -r line; do 
    echo "$line"|sed -e 's/B2vpnId=""/B2vpnId="'"vpn$i"'"/'
    ((i+=1))
done

However, if you need to echo also the lines that will not be changed, you should probably put the grep in the loop.

i=1
while read -r line; do 
    if echo "$line" | grep -q 'B2vpnId=' ; then
        echo "$line"|sed -e 's/B2vpnId=""/B2vpnId="'"vpn$i"'"/'
        ((i+=1))
    else
        echo "$line"
    fi
done < test

This is working from your own solution.

In your last example, you could avoid the grep and the additional else condition by doing something such as: echo "$line" | sed -e '/B2vpnId=/{s/B2vpnId=""/B2vpnId="'"vpn$i"'"/; q0}; q1' && ((i+=1)). Sed will return 0 if B2vpnId= appears in the line and increment the $i, and will return 1 if it didn't match the condition without incrementing $i. — aviro, Nov 30 '21 at 12:59

enumerate all matches in file with sed or perl

3 Answers3