Replace string within multiple files using per line strings from one file

Question

In all the many years using this site I’ve never had to ask a question because there has ALWAYS been an answer (usually numerous). I’m pretty sure this one Does too but for the life of me I cannot find it.

I have directory with a bunch of files which have numerous lines of random length.

a.txt
b.txt
c.txt
d.txt

Then I have single fileeg.txtwith a set list of strings

opq  111
rst  222
uvw  333
xyz  444

Each of the txt files has a single string I’d like to replace

a.txt has a#P#b
b.txt has c#P#d
c.txt has e#P#f
d.txt has g#P#h

I want to replace #P# with the second ‘column’ from my file of strings. The #P# occurs only one time per file (because I’ve put it there). The result would be

a.txt has a111b
b.txt has c222d
c.txt has e333f
d.txt has g444h

The ‘constant‘ assumption is that there are as many lines ineg.txtas there are.txtfiles in my directory and they are in Alphabetical order. The lines ineg.txtare sorted alphabetically as per ‘column’ 1

I’ve been trying to do it using awk and sed (well actually sd) within a for loop but I’m failing to get it to read both ‘source’ and ‘target’ line by line.

I’m not fussy as to how I achieve the result. Currently I’m not working with many lines or files (15 lines and 15 files right now) but there will be times where there will be quite a lot more. I am using zsh as my shell on both an Arch & Debian based linux distro (WSL 2 at times)

Apologies if this has an answer. I’ve really tried to find it over the last two days while working on this project and my brain is now spent.

EDIT: Updated to clarify that the files in the directory have numerous lines of various length and that my given string #P# occurs only once per file

Answer 1

Using GNU awk for "inplace" editing and ARGIND:

awk -i inplace '
    NR == FNR { map[NR]=$2 }
    NR != FNR { sub(/#P#/,map[ARGIND]) }
1' eg.txt ?.txt

The above assumes the replacement text from eg.txt doesn't contain spaces or &s.

Answer 2

preparations

Only one line in each file.

$ grep -- . ?.txt
a.txt:a#P#b
b.txt:c#P#d
c.txt:e#P#f
d.txt:g#P#h

$ cat input
opq  111
rst  222
uvw  333
xyz  444

solution

Have a shell loop call sed for each file:

for file in ?.txt; do
    read -r dummy new_string rest
    sed -- "s/#P#/$new_string/g" "$file"
done <input
a111b
c222d
e333f
g444h

Change that to sed -i with GNU sed or compatible or sed -i '' with FreeBSD sed or compatible if you are satisfied with the result for having the files changed.

The above assumes the lines of input don't contain &, /, nor \ characters. If they may you would have to escape those with backslashes first.

Answer 3

#!/bin/sh
mv eg.txt eg.input
awk 'NR==FNR{a[++i]=$2;next}{sub("#P#",a[++j]);print>(FILENAME".new")}' eg.input ./*.txt &&
for f in *.txt; do mv "$f.new" "$f"; done
mv eg.input eg.txt

eg.txt is renamed to eg.input and then back so that *.txt in the awk line expands only to the files that should be modified.

NR==FNR{    #For the first file, eg.input
  a[++i]=$2   #Put the second field in the array `a`
  next        #Skip the rest of the code
}
{                        #For the other files
  sub("#P#",a[++j])        #Make the substitution
  print>(FILENAME".new")   #Print to the line to `FILENAME`.new
}

Then, in a for loop, the old *.txt files contents are overwritten by the *.new files contents. You may want to suppress the for loop until you are convinced that the *.new files are correct.

---

Some awk implementations do not handle many open files (GNU awk does). If your awk exits with "too many open files" error, use this variant,

awk 'NR==FNR{a[++i]=$2;next}FNR==1{close(fn);fn=FILENAME".new"}{sub("#P#",a[++j]);print>fn}'

Answer 4

Since you already are on zsh and I presume you are with GNU's version of sed, then we can do it like as shown in a two step process.

setopt extended_glob
sed -Ei -e '/#P#/R eg.txt' ./(^eg).txt
sed -Ei -e '/#P#/N;s/#P#(.)\n.\s(.*)/\2\1/' ./(^eg).txt

Brief explanation

• Turn on extended globbing so that we can filter out a specific file eg.txt from the sed commandline.

• Place the respective line from eg.txt after the #P# containing line with the help of the R command. Read up on this GNU specific command in the manual for more info.

• Here we merge the two lines and do a cut n paste job to get the desired output.

The files were edited inplace (except eg.txt)

Answer 5

eg.txt

opq  111
rst  222
uvw  333
xyz  444

a.txt

a#P#b
12345
apple

b.txt

c#P#d
56788

command

j=1;for i in "a.txt" "b.txt" ; do  b=`sed -n ''$j'p' eg.txt| awk '{print $2}'`;sed "s/#P#/$b/g" $i;echo "=================";j=$(($j+1)); done
output
below are the output of a.txt
a111b
12345
apple
=================
below are the output of b.txt
c222d
56788
=================