sed not substituting variable in while loop

Question

When i try to use line number from variable in sed it works individually but when same is done using while loop it not works.

eg.

cat file
den 
run
ten
run
fan
linenumber=2
replacetext=34
sed -i ""$linenumber"s/run/run\n"$replacetext"/" file
Output:
cat file
den
run
34
ten
run
fan

Above works and sed command is generated after variable expansion.

But when i use same in while loop it does work only for first instance and not all as i wanted to replace all instances with incrementing line number using pure sed solution. Awk or perl solution is not wanted.

eg.

replacetext=1
while read in; do replacetext=$((replacetext++)) && sed ""$in"s/run/run\n"$replacetext"/" file; done <<< "$(cat file | grep 'run' -n | cut -f1 -d ':')"

Expected Output: Match text will be replaced with itself followed by newline and a incrementing number.

den 
run
2
ten
run
3
fan

Obtained output:

den
run
2
ten
run
fan

Please only sed based solution required. And i wanted to know what unusual behaviour was occuring in while loop that it was not working.Because the loop variable is given the value of line number and then that line number is used in sed to replace macthed text on that line with itself folowed by newline and incremented number.

Sed seems to work when same variable line number is provided but not when used in while loop.

@Quasímodo is close. (I thing i minor bug in there comment). — ctrl-alt-delor, Jun 27 '20 at 11:51
Dollar expansion happens in double quotes, so no need to close and re-open. — ctrl-alt-delor, Jun 27 '20 at 11:52
If you have GNU sed, you might consider doing something like seq 1 "$(wc -l <file)" | sed -i '/run/R/dev/stdin' file instead. The seq just generates a suitably large number of indices and the sed reads as many of them as it needs. — steeldriver, Jun 27 '20 at 12:16
The R command is however a GNU extension. It fails in BusyBox, for example. — Paul_Pedant, Jun 27 '20 at 12:47
Please note that doing string expansion outside quotation is unnecessary or even error prone. Just use curly braces: sed -i "${linenumber}s/run/run\n${replacetext}/" file. It also increases readability. — rexkogitans, Jun 27 '20 at 16:02
You should copy/paste your script into http://shellcheck.net and fix the issues it tells you about. I think you'll be surprised at how many there are. See also why-is-using-a-shell-loop-to-process-text-considered-bad-practice for why not to do this at all and just use a small awk script instead. — Ed Morton, Jun 27 '20 at 19:52

Paul_Pedant · Accepted Answer · 2020-06-28T20:59:57.260

It fails because the stuff after <<< gets a list of the correct line numbers in the original file.

Then you edit the file multiple times, inserting extra lines.

So every sed except the first is being told the wrong line numbers for the s/// command.

One solution would be to reverse the list of line numbers, so edits did not affect earlier numbering.

This is a sed solution, in one pipeline, somewhat quirky.

The first sed gets the line numbers where the Pattern is found, using the = operator. The nl prefixes a sequence number to each line: like 3 10

The second sed outputs sed expressions: it converts each such line to your required sed edit, like: 10 s/$/\n3/.

The third sed collects and applies those edits from stdin, and does an in-situ edit of the file. In this case, it edits itself.

This is the script itself:

#! /bin/bash
Pattern="[Ss]ed"
RegEx='[^0-9]([0-9])[^0-9]([0-9]).*'
DoSed='\2 s/$/\n\1/'
sed -n -e "/${Pattern}/=" badSed |
    nl -ba |
    sed -e "s:${RegEx}:${DoSed}:" |
    sed -i -f - badSed

This is what it does to itself:

#! /bin/bash
Pattern="[Ss]ed"
RegEx='[^0-9]([0-9])[^0-9]([0-9]).*'
DoSed='\2 s/$/\n\1/'
1
sed -n -e "/${Pattern}/=" badSed |
2
    nl -ba |
    sed -e "s:${RegEx}:${DoSed}:" |
3
    sed -i -f - badSed
4

That's a trick I learned 40 years ago when fixing compiler errors, which show line numbers. If you start editing at the bottom of the error list, the source line numbers match the error messages all the way up. But you get to check a lot of knock-on errors that way. — Paul_Pedant, Jun 28 '20 at 13:22
It is still a shed-load easier in the proper tool: awk '{ print; if (/run/) print ++n } One process, no shell loops, ten seconds to write. — Paul_Pedant, Jun 28 '20 at 13:29

score 0 · Answer 2 · answered Jun 27 '20 at 14:55

0

The line numbers are being messed up where the substitution needs to happen.

So we devise another strategy, we allow the while loop to first generate a valid set of sed commands, which are then applied back to the input file.

replacetext=1
while read in; do ((replacetext++)); echo "$in s/\$/\\n$replacetext/"; done <<<"$(grep -n run file | cut -d: -f1)" | sed -f - file > tmp && mv tmp file

answered Jun 27 '20 at 14:55

Rakesh Sharma

1,102

If you don't use sed -i then you will see duplicated output. Because the sed is invoked as many times as the number of lines comprising the string "run". – Rakesh Sharma Jun 27 '20 at 19:34
Cut off the last part (the sed itself) and just inspect the sed commands that are coming down the pipe. – Paul_Pedant Jun 28 '20 at 13:16
In this case, you are only running one sed, so the -i option will work properly here. The problem with it previously was that the file was altered between each iteration. – Paul_Pedant Jun 28 '20 at 13:18

sed not substituting variable in while loop

2 Answers2