Read file line by line with awk to replace characters in certain line numbers

Question

I have got this script here. It is supposed to run the loop by reading the file LineNumbers.file line by line (each contains a line number) and then accordingly replacing 0/0 with ./. in the BEFORE_File.txt. It works, but it takes only the very last line of the file LineNumbers.file, instead of the >100 entries.

I am not sure what I am doing wrong here. Can you please help me to get awk to read the LineNumbers.file line by line?

I have got it to work with sed -i "${line}s/0\/0/\.\/\./" "${myFileTmp}", but it was really slow on the >3GB large files that I have got. So I thought that awk would be a faster option.

Many thanks!

cat ./LineNumbers_TEMP/LineNumbers.file | while read line
do
myFileTmp=BEFORE_File.txt
awk -v var=${line} 'FNR==var { sub(/0\/0/, "\.\/\."); print }' "${myFileTmp}" > AFTER_File.txt
done

For example this is how the files look like:

cat ./LineNumbers_TEMP/LineNumbers.file
1
2
5

File.txt before script:

cat BEFORE_File.txt
0/0
0/0
0/1
0/1
0/0
0/0
0/0

This is how the file should look after running the script:

cat AFTER_File.txt
./.
./.
0/1
0/1
./.
0/0
0/0

At the moment I get only this:

./.

Answer 1

Your code does not work because for each line number read from LineNumbers.file, you modify the original BEFORE_File.txt and create AFTER_File.txt. Therefore, the final AFTER_File.txt will only contain the changes made to the last line number listed in LineNumbers.file.

Additionally, parsing a whole file just to change a single line, and then doing that many times, is extremely inefficient, and doubly so when the modification to the lines is identical.

It would be better to first read the line numbers, and then to modify all lines in one go:

awk 'FNR == NR { lineno[$1] = 1; next }
     (FNR in lineno) && $0 == "0/0" { $0 = "./." }
     { print }' LineNumbers.file BEFORE_File.txt >AFTER_File.txt

FNR and NR are two special variables in awk that holds the current record number (line number by default) of the current file, and the over all number of records (lines) read so far. For the first input file, these two values will be the same, and when they are, we store the line number as a key in the associative array lineno, and then skip to the next line.

When they are not the same, we test whether the current line number is a key in the lineno array, and whether the current line is additionally equal to 0/0. If so, it is changed to ./.. The last { print } block outputs all lines of the second file, whether modified or not.

---

A totally different approach is to use sed to create a sed script to do the necessary changes.

Given a line number n, the sed expression ns,^0/0$,./., would change line n by substituting 0/0 into ./.. If the line is anything but exactly 0/0, no changes would be made. I'm using commas as delimiters for the s/// command to avoid the leaning toothpick syndrome.

All we have to do is to create expressions like that for each line number n:

sed 's#.*#&s,^0/0$,./.,#' LineNumbers.file

Here, I use # as delimiters for s///. The & in the replacement part of the command would be replaced by the line number read from the input file.

For the given list of line numbers, this generates

1s,^0/0$,./.,
2s,^0/0$,./.,
5s,^0/0$,./.,

We can simply apply this directly to our file:

sed 's#.*#&s,^0/0$,./.,#' LineNumbers.file | sed -f /dev/stdin BEFORE_File.txt >AFTER_File.txt

Answer 2

Let's see if this works for you:

awk '{ 
  if ( NR == FNR ) { 
    n[$1] = 0 
  } else { 
    if ( FNR in n ) { 
      gsub(/^0\/0$/, "./.", $0) 
    } 
    print 
  } 
}' LineNumbers.file BEFORE_File.txt > AFTER_File.txt

Output:

./.
./.
0/1
0/1
./.
0/0
0/0

Answer 3

Given that your input actually looks like blabla 4858 ABC 0/0:4,3,2 0/1:4,3,2 rather than the example you posted in your question, all you need is:

awk 'NR==FNR{a[$1]; next} FNR in a{sub("0/0","./.")} 1' LineNumbers.file BEFORE_File.txt >AFTER_File.txt

For your next question please post sample input that looks like your real input to avoid getting answers that are too simple or more complicated than necessary and/or only work with input you don't actually have.

Don't do this as it's a bad approach in multiple ways but FYI if you were to use a shell loop like in your question then you'd write it as:

myFileTmp=$(mktemp)
cp BEFORE_File.txt AFTER_File.txt
while IFS= read -r line
do
    awk -v var="${line}" '
        FNR==var { sub("0/0", "./.") } { print }
    ' AFTER_File.txt > "$myFileTmp" &&
    mv "$myFileTmp" AFTER_File.txt
done < LineNumbers.file

Also wrt the script in your question - "\.\/\." in your gsub() is a string. You don't need to escape regexp metacharacters in strings, just in regexps. Ditto for /. All you need to write there is "./.". See Why is using a shell loop to process text considered bad practice?, http://porkmail.org/era/unix/award.html, and https://mywiki.wooledge.org/Quotes for some of the other issues with your script beyond just the problem you're having.

Answer 4

There is the possibility of reading the lines from the file with the line numbers directly into a variable in awk via getline (assuming line numbers are sorted):

getline var <"filename"

The whole script will be one single call to awk as follows:

awk -v f1='./LineNumbers.file' '
       NR >var+0 {    rc=getline var <f1;
                      if(rc<0){  stderr = "cat 1>&2";
                                 print "error reading",f1 | stderr;
                                 close(stderr);
                                 exit 1
                              }
                 }
       NR==var+0 {    sub(/0\/0/,"./.")
                 }
     1' BEFORE_File.txt

Of course, redirect the output to any file you like.