sed - replace value to use quotes where needed

Question

I have 100+ jinja template files with 0-k occurrences of "value: ..." string in each of them. The problem is that some of the files are using:

value: something

some of them:

value: 'something'

and some of them:

value: "some other thing"

I need all of these to look the same, to use double quotes. I thought I'd do it with sed:

sed -i 's/value: ['"]?(.*)['"]?/value: "\1"/g' *.j2

but as you can see I'm quite horrible with sed and the past 2 hours only made me want to break my keyboard with the nonsense error messages I'm getting, like: unterminated `s' command and such.

Sample input:

- param:
  name: Command
  type: String
  value: '/bin/echo'
- param:
  name: Args
  type: String
  value: Hello World
- param:
  name: Something
  type: EnvVar
  value: "PATH"

from this I need to get:

- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

Answer 1

When you need to use the two forms of quotes ("') in the expression, things get tricky. For one, in your original attempt the shell identifies this 's/value: [' as a quoted string: the latter quote is not preserved.

In these cases, rather than having a headache, you can simply put the Sed commands in a file. Its contents won't be subject to the shell manipulation.

quotes.sed:

# (1) If line matches this regex (value: '), 
# (2) substitute the first ' with " and
# (3) substitute the ' in the end-of-line with ".
/value: '/{
  s/'/"/
  s/'$/"/
}
# (4) If line matches this regex (value: [^"]), 
# (5) substitute :<space> for :<space>" and
# (6) append a " in the end-of-line.
/value: [^"]/{
  s/: /: "/
  s/$/"/
}

$ sed -Ef quotes.sed file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

Answer 2

The problem with your sed call is two-fold

• You have literal single quotes as part of your regular expression ['"] in a sed program enclosed in single quotes. This will not work as single quotes inside single quotes cannot be escaped. You can represent them as \x27, however.
• The ... (.*) ... syntax is extended regular expression syntax for having ( and ) mean "capture group" by default. Your should either use the -E option of sed, or define the capture group as \(.*\) to use basic regular expressions (but then you would need to replace the ? with something like \{0,1\}, too).

The following worked in my tests:

sed -E 's/value: [\x27"]?([^\x27"]*)[\x27"]?/value: "\1"/' input.j2

Answer 3

With GNU awk for the 3rd arg to match() and gensub():

$ awk 'match($0,/(\s*value:\s*)(.*)/,a){$0=a[1] "\"" gensub(/[\047"]/,"","g",a[2]) "\""} 1' file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

otherwise Using any awk:

$ awk '/value:/{hd=$0; sub(/:.*/,": ",hd); gsub(/[^:]*: *[\047"]?|[\047"]$/,"");  $0=hd "\"" $0 "\""} 1' file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

Answer 4

Using Raku (formerly known as Perl6)

A gentleman by the name of Larry Wall posted this neat quoting tip on the "perl6-users" mailing list (perl6-users@perl.org). See Tip #8 at the NNTP link below:

~$ env re="\'" raku -ne 'say $0 if m/ <?after value\:\s >  \"? <{ %*ENV<re> }>?  (.+?) \"? <{ %*ENV<re> }>? $$ /;'  jinja.txt
｢/bin/echo｣
｢Hello World｣
｢PATH｣

The code above shows the correct match. So all that's needed is to run the code below:

~$ env re="\'" raku -pe 's/ <?after value\:\s >  \"? <{ %*ENV<re> }>?  (.+?) \"? <{ %*ENV<re> }>? $$ /"{$0}"/;'  jinja.txt
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

Want to make your escaping more readable? Don't want to play with ENV variables as above? Then just spell out your quoting characters (also from Larry Wall's Tip #8 at the NNTP link below):

~$ raku -ne 'say $0 if m/ <?after value\:\s >  \c[QUOTATION MARK]? \c[APOSTROPHE]?  (.+?)  \c[APOSTROPHE]? \c[QUOTATION MARK]? $$ /;'  jinja.txt
｢/bin/echo｣
｢Hello World｣
｢PATH｣
~$ raku -pe 's/ <?after value:\s >  \c[QUOTATION MARK]? \c[APOSTROPHE]?  (.+?)  \c[APOSTROPHE]? \c[QUOTATION MARK]? $$ /"{$0}"/;'  jinja.txt

param:
name: Command
type: String
value: "/bin/echo"
param:
name: Args
type: String
value: "Hello World"
param:
name: Something
type: EnvVar
value: "PATH"

HTH.

https://raku.org/
https://www.nntp.perl.org/group/perl.perl6.users/2020/07/msg9004.html