2

I have a CSV file (say temp.csv) with strings and numbers. I need to delete the lines which contain a particular pattern of string at the beginning of the lines in the file. For example, here is my file - 

req1,incl_patt1,excl_patt1,2,ind1
req1,incl_patt2,excl_patt2,1,ind1
req1,incl_patt3,excl_patt3,4,ind1
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

I am writing a pattern to req_file_curr in a command ~ $ req_file_curr=req1

echo "${req_file_curr}" gives output as req1

However when using it in a sed command, like below:

sed '/\"${req_file_curr}\"/d' temp.csv

gives output as

req1,incl_patt1,excl_patt1,2,ind1
req1,incl_patt2,excl_patt2,1,ind1
req1,incl_patt3,excl_patt3,4,ind1
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

I am expeceting the output as 

req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

How can I use the variable from a shell as a pattern to look for in sed or awk?

Gilles 'SO- stop being evil'
  • 829,060
Dhruuv
  • 577

4 Answers4

6

You need to enclose the sed pattern in double quotes so that the shell can expand the variable before passing it to sed:

sed "/^${req_file}/d" temp.csv

You also need to use the ^ anchor to indicate to sed that this pattern needs to occur at the beginning of the line.

Joseph R.
  • 39,549
5

No need for awk or sed here:

grep -v "^$req_file_curr," file

Note that (like for sed or awk though to a lesser extent), that assumes that $req_file_curr does not contain regexp operators (like ., *...).

If it does, you'd have to escape those characters, or with awk, you could use an approach using its index() function:

S="$req_file_curr," awk 'index($0, ENVIRON["S"]) != 1' < file
Stéphane Chazelas
  • 544,893
  • I tried to append .req to the contents of first columns to check if sed or awk works, it actually works fine with the code provided by @joseph. req_file_curr=req1.req. Using the sed command as provided above sed "/^${req_file}/d" temp.csv gives exactly what I need. Am I missing anything here? – Dhruuv May 13 '14 at 16:36
  • @Dhruuv, . matches any character, so req1.req would match req1.req but also req12req or req1,req. – Stéphane Chazelas May 13 '14 at 17:05
  • I do have . in the first column of my file. I am leaning more towards sed because I can use inplace edit option sed -i (which I learned recently) instead of streaming the output into a file and then renaming that file to the original file name. Is there a way to handle the regexp operators while using sed? Otherwise, I will have to go ahead with grep – Dhruuv May 13 '14 at 17:55
  • 1
    @Dhruuv See http://unix.stackexchange.com/a/129063/22565 (escape as per the LHS) – Stéphane Chazelas May 13 '14 at 19:14
  • grep also does regular expressions. awk has an index() function that compares actual strings, not using patterns. – Paul_Pedant Jan 14 '20 at 13:07
  • @Paul_Pedant, good point thanks. I've added an index()-based appraoch. – Stéphane Chazelas Jan 14 '20 at 13:24
4

You have set req_file_curr but use req_file. And your sed command won't work, since shell variable can not be expanded in your sed command.

You can try this:

$ sed '/'"${req_file_curr}"'/d' file
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

or using sed with double quote:

sed "/${req_file_curr}/d" file

and using awk:

$ awk -v req=$req_file_curr '$0 !~ req' file
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4
cuonglm
  • 153,898
0

An ed solution would be something like.

  req_file_curr=req1

  printf '%s\n' "g/^$req_file_curr/d"  w | ed -s temp.csv

It edit's the file inplace so test first before running this against your files.

Jetchisel
  • 1,264