12

I have a set of sed replacements in a bash script and am getting an error about an unterminated `s' command. Here's what the sed line looks like: 

sed -n -e "s/TMPFOO1/$FOO1/" -e "s/TMPFOO2/$FOO2/" -e "s/TMPFOO3/$FOO3/" -e "s/TMPFOO4/$FOO4/" -e "s/TMPFOO5/$FOO5/" /home/foo/template > /home/foo/template/finishedresult

For some reason though, bash doesn't like this and I get an error about 

sed: -e expression #4, char 69: unterminated `s' command

What am I missing here? How can I have SED input the variable? It looks to me like they are all terminated.

Gilles 'SO- stop being evil'
  • 829,060
Mike B
  • 8,900
  • 2
    $FOO4 contains something weird (ex: "" or "\" ?) that makes the sed s command not terminated. (That's why I think it's a trailing "", making the next "/" be treated as a character part of the replace string, instead of as the terminating character for the s command) – Olivier Dulac May 10 '13 at 07:42

4 Answers4

15

You cannot safely interpolate a variable in a sed command, because the substitution is performed by the shell, not by sed. The value of the variable becomes sed syntax. For example, in "s/TMPFOO1/$FOO1/", if $FOO1 contains a newline, this will cause a syntax error like the one you observed. If $FOO1 contains a /, this terminates the s command and may cause an error or may cause other commands to be executed (if what's after the / happens to be valid sed syntax).

While you can do a first pass of substitution of FOO1 to quote its special characters for inclusion in that sed command, it's a lot simpler to use awk. Awk has a notion of variable, and a command line syntax to set the initial value of a variable.

awk -v FOO1="$FOO1" -v FOO2="$FOO2" -v FOO3="$FOO3" -v FOO4="$FOO4" -v FOO5="$FOO5" '{
    sub(/TMPFOO1/, FOO1);
    sub(/TMPFOO2/, FOO2);
    sub(/TMPFOO3/, FOO3);
    sub(/TMPFOO4/, FOO4);
    sub(/TMPFOO5/, FOO5);
    print;
}' /home/foo/template > /home/foo/template/finishedresult
Gilles 'SO- stop being evil'
  • 829,060
  • Trivial note: - a -v needs to precede each variable declaration. – Mike B Feb 06 '14 at 06:01
  • @MikeB Thanks for the bug report. In general, please edit the post directly when you find typos like this. – Gilles 'SO- stop being evil' Feb 06 '14 at 10:13
  • @Gilles Edits from non-authors must be at least six characters. – NobleUplift Feb 06 '14 at 22:09
  • What does it help if the $FOO1 has a \n in it. Init it will not remove the \n. I read line by line in bash script and replace some lines like sed "s/$li/- [x]${li:5}/" $dat .Should I stay with the script and use awk, or find out how to work with \n in sed or better move to e.g.python? – Timo Nov 18 '20 at 18:42
  • 1
    @Timo I don't understand the first two sentences of your comment. If you want to replace strings containing newlines, sed is very hard to work with. Awk is easier: set a value of FS that doesn't match anything that $FOO1 matches. – Gilles 'SO- stop being evil' Nov 18 '20 at 20:11
  • Thanks Gilles, here is the sh code with sed line 42, here is the py code. The basis here. As you see, text is read that can have special characters as it is code (angular). Maybe not use awk but stay with py? – Timo Nov 19 '20 at 08:08
  • 1
    @Timo I can't answer that in a comment. Ask a new question. – Gilles 'SO- stop being evil' Nov 19 '20 at 08:34
  • Bewary of sed's special characters that need to be escaped. See also: https://unix.stackexchange.com/questions/32907/what-characters-do-i-need-to-escape-when-using-sed-in-a-sh-script – randomScott Nov 23 '21 at 01:05
6

Most likely one of your $FOO variables contains special characters that are interpreted by sed.

I have another version of sed which generates other error messages but here is an example of a similar problem:

$ VAR=a
$ echo i | sed -e "s/i/"$VAR"/"
a
$ tmp> VAR=/
$ echo i | sed -e "s/i/"$VAR"/"
sed: 1: "s/i///
": bad flag in substitute command: '/'

In this case $VAR contains a character which is interpreted by sed as the trailing slash.

Matteo
  • 9,796
  • 4
  • 51
  • 66
2

As others have mentioned here, it depends on the content of your FOO* variables. In your case, using sed is the wrong choice because it probably contain some special characters.

Take a look at this link and see the function gsub_literal

Since gsub_literal reads from stdin and writes to stdout, the usage will be:

gsub_literal "$search" "$replace" < /home/foo/template > /home/foo/template/finishedresult

Sample output:

rany$ cat > foo.txt
a'
a'

rany$ gsub_literal a\' b < foo.txt 
b
b
Rany Albeg Wein
  • 670
1

The problem has been resolved for me by trying additional backslashes within the sed 's/.../.../g' command. This command 

sed 's/\"a/ä/g' input_file

gave me this error message

sed: -e expression #1, char 9: unterminated `s' command

when it was executed as part of a c-shell script, and it did so only with a recent version of sed, not with an older version. (Strangely, the same command works fine when it is given on the command line (in a tc-shell)).

The problem with the c-shell script has been remedied by preceding the ä character by a backslash:

sed 's/\"a/\ä/g' input_file
jasonwryan
  • 73,126
user3346151
  • 11