How to enclose variables with double quotes correctly in sed command

Question

I have a .csv file with two "columns" of data like this:

test1.ts.meta,Before Sunrise (1995)
test2.ts.meta,A Beautiful Mind (2001)
test3.ts.meta,Departures (2008)
test4.ts.meta,Love & Other Drugs (2010)

I am trying to use this command to replace line 2 in each .ts.meta file with the corresponding movie name...

cat 1TBMovie2_dotTSdotMeta.txt | while IFS=, read file moviename; do sed "2 s/^.*$/$moviename/" "$file"; done

It works fine except for movie names that have the ampersand in them (&).
For example the movie name: Love & Other Drugs (2010) in this case line two of the .ts.meta file gets this movie name:

Love Love Love & Other Drugs Other Drugs (2010) Other Drugs (2010)

Likewise the movie name: Love & Mercy (2015) appears in the .ts.meta file like: Love Love Love & Mercy Mercy (2015) Mercy (2015).

Confusingly... If I open the .ts.meta file for a movie called Love & Mercy (2015) and manually delete line 2 and save then run the command above again I get this in line 2... Love Mercy (2015) with two spaces between the words "Love" and "Mercy".

I think I need to enclose the $moviename variable in double quotes like I did with the $file variable? I guess the ampersand character is being read by sed as having special meaning?

Here is some more information to clarify the problem

My csv file (I actually call it: updatemeta.txt

test1.ts.meta,Carols from King's (2013)
test2.ts.meta,Before Sunrise (1995)
test3.ts.meta,Love & Other Drugs (2010)
test4.ts.meta,Departures (2008)

test1.ts.meta

1:0:19:1B1C:802:2:11A0000:0:0:0:
Carols from King's
The traditional Christmas carol concert from King's College Chapel, Cambridge. Stephen Cleobury conducts the famous chapel choir in carols old and new. [HD] [S]
1387969020
448066800
2913369072
f:0,c:00157c,c:01157e,c:02157f,c:03157c,c:050001
188
0

test2.ts.meta

1:0:1:189E:7FD:2:11A0000:0:0:0:
Before Sunrise
Romance starring Julie Delpy and Ethan Hawke. Two twentysomethings meet on a train and decide to spend a few hours together. Contains some strong language.  Also in HD. [1995] [AD,S]
1392418980
550744512
2637755808
f:0,c:0013ec,c:0113ed,c:0213ef,c:0313ec
188
0

test3.ts.meta

1:0:1:2404:7F9:2:11A0000:0:0:0:
Love & Other Drugs
(2010) Fact-based adult comedy. Jake Gyllenhaal stars as a successful Viagra salesman who falls for a woman with Parkinson's (Anne Hathaway). Strong language/sexual scenes.  [AD,S]
1472775840
712401799
2824257448
f:0,c:000931,c:010932,c:020934,c:030931
188
0

test4.ts.meta

1:0:1:2404:7F9:2:11A0000:0:0:0:
Departures
(2008) An Oscar-winning, whimsical look at the Japanese undertaking profession. Masahiro Motoki stars as a musician starting a new career preparing the dead for burial. Japanese/subs.
1400111580
863881200
3699150040
f:0,c:000931,c:010932,c:020934,c:030931
188
0

I will have the .csv file in the same directory as many .ts.meta files. There will be a row in the .csv file for each .ts.meta file in the directory and a corresponding movie name.

How do I use sed or awk or gawk to create a script that loops through each row in the .csv file and replaces line two in the named .ts.meta file with the corresponding movie name specified in the .csv file?

I tried the examples given in the solutions below but don't understand what is going on!

Thank you,

Flex

Related: What characters do I need to escape when using sed in a sh script? - in particular, note that & is a special character on the replacement side of a sed substitution — steeldriver, Jul 08 '20 at 00:31

Ed Morton · Accepted Answer · 2020-07-08T21:40:13.213

Don't write shell loops just to manipulate text, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice, and when you want to use literal strings use a tool like awk that understands literal strings, not a tool like sed that doesn't.

You didn't provide any .ts.meta files for us to test against so obviously this is untested but something like this will do the job using GNU awk for -i inplace (assuming you want to modify the original files) and ARGIND:

awk -i inplace -F',' '
    NR == FNR {
        titles[ARGC] = $2
        ARGV[ARGC++] = $1
    }
    (NR != FNR) && (FNR == 2) {
        $0 = titles[ARGIND]
    }
    { print }
' 1TBMovie2_dotTSdotMeta.txt

If you REALLY wanted to try to do this with sed (don't!) then see is-it-possible-to-escape-regex-metacharacters-reliably-with-sed and note that & isn't the only character you'd have to worry about, e.g. / and \1, for example, would also need to be handled.

Given your newly provided sample input:

$ head -n 50 update* *.meta
==> updatemeta.txt <==
test1.ts.meta,Carols from King's (2013)
test2.ts.meta,Before Sunrise (1995)
test3.ts.meta,Love & Other Drugs (2010)
test4.ts.meta,Departures (2008)
==> test1.ts.meta <==
1:0:19:1B1C:802:2:11A0000:0:0:0:
Carols from King's
The traditional Christmas carol concert from King's College Chapel, Cambridge. Stephen Cleobury conducts the famous chapel choir in carols old and new. [HD] [S]
1387969020
448066800
2913369072
f:0,c:00157c,c:01157e,c:02157f,c:03157c,c:050001
188
0
==> test2.ts.meta <==
1:0:1:189E:7FD:2:11A0000:0:0:0:
Before Sunrise
Romance starring Julie Delpy and Ethan Hawke. Two twentysomethings meet on a train and decide to spend a few hours together. Contains some strong language.  Also in HD. [1995] [AD,S]
1392418980
550744512
2637755808
f:0,c:0013ec,c:0113ed,c:0213ef,c:0313ec
188
0
==> test3.ts.meta <==
1:0:1:2404:7F9:2:11A0000:0:0:0:
Love & Other Drugs
(2010) Fact-based adult comedy. Jake Gyllenhaal stars as a successful Viagra salesman who falls for a woman with Parkinson's (Anne Hathaway). Strong language/sexual scenes.  [AD,S]
1472775840
712401799
2824257448
f:0,c:000931,c:010932,c:020934,c:030931
188
0
==> test4.ts.meta <==
1:0:1:2404:7F9:2:11A0000:0:0:0:
Departures
(2008) An Oscar-winning, whimsical look at the Japanese undertaking profession. Masahiro Motoki stars as a musician starting a new career preparing the dead for burial. Japanese/subs.
1400111580
863881200
3699150040
f:0,c:000931,c:010932,c:020934,c:030931
188
0

Here's the awk script running:

$ awk -i inplace -F',' '
    NR == FNR {
        titles[ARGC] = $2
        ARGV[ARGC++] = $1
    }
    (NR != FNR) && (FNR == 2) {
        $0 = titles[ARGIND]
    }
    { print }
' updatemeta.txt

and here's what that did to your files:

$ head -n 50 update* *.meta
==> updatemeta.txt <==
test1.ts.meta,Carols from King's (2013)
test2.ts.meta,Before Sunrise (1995)
test3.ts.meta,Love & Other Drugs (2010)
test4.ts.meta,Departures (2008)
==> test1.ts.meta <==
1:0:19:1B1C:802:2:11A0000:0:0:0:
Carols from King's (2013)
The traditional Christmas carol concert from King's College Chapel, Cambridge. Stephen Cleobury conducts the famous chapel choir in carols old and new. [HD] [S]
1387969020
448066800
2913369072
f:0,c:00157c,c:01157e,c:02157f,c:03157c,c:050001
188
0
==> test2.ts.meta <==
1:0:1:189E:7FD:2:11A0000:0:0:0:
Before Sunrise (1995)
Romance starring Julie Delpy and Ethan Hawke. Two twentysomethings meet on a train and decide to spend a few hours together. Contains some strong language.  Also in HD. [1995] [AD,S]
1392418980
550744512
2637755808
f:0,c:0013ec,c:0113ed,c:0213ef,c:0313ec
188
0
==> test3.ts.meta <==
1:0:1:2404:7F9:2:11A0000:0:0:0:
Love & Other Drugs (2010)
(2010) Fact-based adult comedy. Jake Gyllenhaal stars as a successful Viagra salesman who falls for a woman with Parkinson's (Anne Hathaway). Strong language/sexual scenes.  [AD,S]
1472775840
712401799
2824257448
f:0,c:000931,c:010932,c:020934,c:030931
188
0
==> test4.ts.meta <==
1:0:1:2404:7F9:2:11A0000:0:0:0:
Departures (2008)
(2008) An Oscar-winning, whimsical look at the Japanese undertaking profession. Masahiro Motoki stars as a musician starting a new career preparing the dead for burial. Japanese/subs.
1400111580
863881200
3699150040
f:0,c:000931,c:010932,c:020934,c:030931
188
0

Thanks for your answer but I'm still lost! I updated my question to include example *.ts.meta files. When I tried your awk solution I was getting ^ invalid char ''' in expression pointing at the apostrophe in the movie name: Carols from King's and ^ syntax error pointing at: test2.t.meta. Can you make your code into a script that uses my additional information please? Thank you. — FlexMcMurphy, Jul 08 '20 at 21:25
@FlexMcMurphy I fixed a bug in the script and updated my answer to show it running with the sample input you've now provided. — Ed Morton, Jul 08 '20 at 21:40
That's brilliant thank's for your help, all working now. I need to study up on awk which I know nothing about ! I'm accepting your answer because as you say it seems that awk is the better tool for this job. — FlexMcMurphy, Jul 08 '20 at 23:08
I highly recommend the book Effective AWK Programming, 4th Edition, by Arnold Robbins. Don't get any other book as they're all less useful and mostly out of date. — Ed Morton, Jul 09 '20 at 12:02

Rakesh Sharma · Answer 2 · 2020-07-08T21:58:20.193

1

One way to do that is to bypass the regex route and use the read r command of sed.

cat 1TBMovie2_dotTSdotMeta.txt | while IFS=, read file moviename; do printf '%s\n' "$moviename" | sed -i -e '2r /dev/stdin' -e '2d' "$file"; done

It should be written soread out on multiple lines as shown below:

cat 1TBMovie2_dotTSdotMeta.txt | 
while IFS=, read file moviename
do
   printf '%s\n' "$moviename" |
   sed -i -e '2r /dev/stdin' -e '2d' "$file"
done

Here we are using gnu sed feature to read file off of stdin. For non-GNU seds we can save the movie name in a temp file and use that name in the r command. Now you don't have to bother escaping anything.

But if you don't want the hassle of an extra file then we need to escape the special chars of / \ & which are special on the rhs of a sed s/.../.../ command. The / is included because it functions as the delimiter.

cat 1TBMovie2_dotTSdotMeta.txt |
while IFS=, read file moviename
do
    moviename_esc=$(printf '%s\n' "$moviename" | sed -e 's:[\&/]:\\&:g')
    sed -i -e "2 s/.*/$moviename_esc/" "$file"
done

edited Jul 08 '20 at 21:58

answered Jul 08 '20 at 06:01

Rakesh Sharma

1,102

Thanks for your answer. I do have GNU sed but I'm not knowledgeable enough about linux to know how to integrate your solution into my problem. I tried the command: cat updatemeta.txt | while IFS=, read file moviename; do printf '%s\n' "$moviename" | sed -e '2r /dev/stdin' -e '2d' "$file"; but all I got was a > prompt at the command line. I updated my question with more information. Can you make your code into a script that uses my additional information please? Thank you! – FlexMcMurphy Jul 08 '20 at 21:29
1

I have updated the answer you may take a look. – Rakesh Sharma Jul 08 '20 at 21:53
cat file | while read; do ... done should instead be written while read; do ... done < file to avoid the UUOC and there's no reason to remove the -r that should be present in all reads by default since you don't need read to interpret escape sequences for you. See also why-is-using-a-shell-loop-to-process-text-considered-bad-practice. – Ed Morton Jul 08 '20 at 22:38
Thanks Rakesh. I was so close to getting your command working I just forgot to add the word --> done <--- at the end! I like your solution because it is easy to understand what's going on. Now I just need to read up some more on the awk command. – FlexMcMurphy Jul 08 '20 at 23:05

How to enclose variables with double quotes correctly in sed command

2 Answers2