Is there a way to prevent sed from interpreting the replacement string?

Question

If you want to replace a keyword with a string using sed, sed tries hard to interpret your replacement string. If the replacement string happens to have characters that sed considers special, like a / character, it will fail, unless of course you meant your replacement string to have characters that tell sed how to act.

Ex:

VAR="hi/"
sed "s/KEYWORD/$VAR/g" somefile

Is there any way to tell sed not to try to interpret the replacement string for special characters? All I want is to be able to replace a keyword in a file with the contents of a variable, no matter what that content is.

Answer 1

There are only 4 special characters in the replacement part: \, &, newline and the delimiter (ref)

$ VAR='abc/def&ghi\foo
next line'

$ repl=$(sed -e 's/[&\\/]/\\&/g; s/$/\\/' -e '$s/\\$//' <<<"$VAR")

$ echo "$repl"
abc\/def\&ghi\\foo\
next line

$ echo ZYX | sed "s/Y/$repl/g"
Zabc/def&ghi\foo
next lineX

Answer 2

You can use Perl instead of sed with -p (assume loop over input) and -e (give program on command line). With Perl you can access environment variables without interpolating these in shell. Note that the variable needs to be exported:

export VAR='hi/'
perl -p -e 's/KEYWORD/$ENV{VAR}/g' somefile

If you do not want to export the variable everywhere, then just provide it for that process only:

PATTERN="$VAR" perl -p -e 's/KEYWORD/$ENV{PATTERN}/g' somefile

Do note, that Perl's regular expression syntax is by default slightly different from sed's.

Answer 3

The very simplest solution which would still handle the vast majority of variable values correctly, would be to use a non-printing character as a delimiter to sed's substitute command.

In vi and in many shells you can escape any control character by typing Ctrl-V (more commonly written as ^V). So if you use some control character (I often use ^A as a delimiter in these cases) then your sed command will only break if that nonprinting character is present in the variable you're dropping in.

So you would type "s^V^AKEYWORD^V^A$VAR^V^Ag" and what you would get (in vi or your shell) would look like:

sed "s^AKEYWORD^A$VAR^Ag" somefile

(You can't copy and paste this from this answer. You have to actually type it as described.)

This will work as long as $VAR doesn't contain the non-printing character ^A—which is exceedingly unlikely.

---

Of course, if you're passing user input into the value of $VAR, then all bets are off and you'd better sanitize your input thoroughly rather than relying on control characters being hard to type for the average user.

---

There is actually more to beware of than the delimiter string, though. For instance, &, when present in a replacement string, means "the entire text that was matched." E.g., s/stu../my&/ would replace "stuff" with "mystuff", "stung" with "mystung", etc. So if you might have any character in the variable that you're dropping in as a replacement string, but you want to use the literal value of the variable only, then you have some data sanitizing to do before you can use the variable as a replacement string in sed. (The data sanitizing can be done with sed also, though.)

Answer 4

You could use a , or a | instead and it will take it as a seperator and technically you could use anything

from the man page

\cregexpc
           Match lines matching the regular expression regexp.  The  c  may
      be any character.

As you can see you should start with a \ before your separator at the beginning ,then you can use it as a separator.

from the documentation http://www.gnu.org/software/sed/manual/sed.html#The-_0022s_0022-Command :

The / characters may be uniformly replaced by any other single character 
within any given s command.

The / character (or whatever other character is used in its stead) can appear in 
the regexp or replacement only if it is preceded by a \ character.

Example:

sed -e 'somevar|s|foo|bar|'
echo "Hello all" | sed "s_all_user_"
echo "Hello all" | sed "s,all,user,"

echo "Hello/ World" | sed "s,Hello/,Neo,"

Answer 5

You can backslash-escape the forward slashes in your replacement string, using Bash's pattern substitution parameter expansion. It's a little messy because the forward slashes also need to be escaped for Bash.

$ var='a/b/c';var="${var//\//\\/}";echo 'this is a test' | sed "s/i/$var/g"

output

tha/b/cs a/b/cs a test

You could put the parameter expansion directly into your sed command:

$ var='a/b/c';echo 'this is a test' | sed "s/i/${var//\//\\/}/g"

but I think the first form is a little more readable. And of course if you're going to re-use the same replacement pattern in multiple sed commands it makes sense to just do the conversion once.

Another option would be to use a script written in awk, perl or Python, or a C program, to do your substitutions instead of using sed.

---

Here's a simple example in Python that works if the keyword to be replaced is a complete line in the input file (not counting the newline). As you can see, it's essentially the same algorithm as your Bash example, but it reads the input file more efficiently.

import sys

#Get the keyword and replacement texts from the command line
keyword, replacement = sys.argv[1:]
for line in sys.stdin:
    #Strip any trailing whitespace
    line = line.rstrip()
    if line == keyword:
        line = replacement
    print(line)

Answer 6

If it's line-based and only one line to replace, I recommend prepending the file itself with the replacement line using printf, storing that first line in sed's hold space, and dropping it in as needed. This way you don't have to worry about special characters at all. (The only assumption here is that $VAR contains a single line of text without any newlines, which is what you said in the comments already.) Other than newlines, VAR could contain anything whatsoever and this would work regardless.

VAR=whatever
{ printf '%s\n' "$VAR";cat somefile; } | sed '1{h;d;};/KEYWORD/g'

printf '%s\n' will print the contents of $VAR as a literal string, regardless of its contents, followed by a newline. (echo will do other things in some cases, for example if the contents of $VAR begins with a hyphen—it will be interpreted as an option flag being passed to echo.)

The braces are used to prepend the output of printf to the contents of somefile as it's passed to sed. Whitespace separating the curly braces by themselves is important here, as is the semicolon before the closing curly brace.

1{h;d;}; as a sed command will store the first line of text in sed's hold space, then delete the line (rather than printing it).

/KEYWORD/ applies the following actions to all lines that contain KEYWORD. The action is get, which gets the contents of the hold space and drops it in place of the pattern space—in other words, the entire current line. (This isn't for replacing only part of a line.) The hold space isn't emptied out, by the way, just copied into the pattern space, replacing whatever is there.

If you want to anchor your regex so it won't match a line which merely contains KEYWORD but only a line where there is nothing else on the line but KEYWORD, add a beginning of line anchor (^) and end of line anchor ($) to your regex:

VAR=whatever
{ printf '%s\n' "$VAR";cat somefile; } | sed '1{h;d;};/^KEYWORD$/g'

Answer 7

If you need to do this in a script, you can go with an escape function, i.e.:

#!/bin/bash
escvar () {
  sed -e 's/[/&]/\&/g' <<< $1
}
replacement='https://google.com/?query=some\delimited|(query)&count=1'
sed -e "s/<placeholder>/$(escvar $replacement)/" <<< 'value=<placeholder>'

It should be enough to escape "dangerous" characters for sed.

Answer 8

This is the way I went:

#Replaces a keyword with a long string
#
#This is normally done with sed, but sed
#tries to interpret the string you are
#replacing the keyword with too hard
#
#stdin - contents to look through
#Arg 1 - keyword to replace
#Arg 2 - what to replace keyword with
replace() {
        KEYWORD="$1"
        REPLACEMENT_STRING="$2"

        while IFS= read -r LINE
        do
                if [[ "$LINE" == "$KEYWORD" ]]
                then
                        printf "%s\n" "$REPLACEMENT_STRING"
                else
                        printf "%s\n" "$LINE"
                fi
        done < /dev/stdin
}

this works great in my case because my keyword is on a line all by itself. If the keyword was in a line with other text, this would not work.

I would still really like to know if there's an easy way to do this that doesn't involve coding my own solution.