67

I want to do:

cat update_via_sed.sh | sed 's/old_name/new_name/' > new_update_via_sed.sh

in my program.

But I want to use variables, e.g.

old_run='old_name_952'
new_run='old_name_953'

I have tried using them but the substitution doesn't happen (no error). I have tried:

cat update_via_sed.sh | sed 's/old_run/new_run/'
cat update_via_sed.sh | sed 's/$old_run/$new_run/'
cat update_via_sed.sh | sed 's/${old_run}/${new_run}/'
don_crissti
  • 82,805
Michael Durrant
  • 42,013

9 Answers9

54

You could do:

sed "s/$old_run/$new_run/" < infile > outfile

But beware that $old_run would be taken as a regular expression and so any special characters that the variable contains, such as / or . would have to be escaped. Similarly, in $new_run, the & and \ characters would need to be treated specially and you would have to escape the / and newline characters in it.

If the content of $old_run or $new_run is not under your control, then it's critical to perform that escaping, or otherwise that code amounts to a code injection vulnerability.

Stéphane Chazelas
  • 544,893
  • 3
    I was using single quote in sed params, switched to double quotes and it worked file. Awesome. – Aakash Jul 28 '16 at 08:30
  • it isn't that sed alone will not use regex but sed -E would ? – Kiwy Apr 24 '18 at 10:04
  • @Kiwy, no. -e is to specify a sed expression, so you can pass more than one (as in sed -e exp1 -e exp2) or combinations of files and expressions (sed -f file -e exp). You may be confusing with -E which with some implementations, tells sed to use EREs instead of BREs. – Stéphane Chazelas Apr 24 '18 at 10:06
  • I mistyped, but OK, that explain why i have so many issues using sed without -E I only master the extended regex not the regular one. Thanks for the explanaiton. – Kiwy Apr 24 '18 at 10:12
  • @Kiwy, see also the linked Q&A for more options supported by some sed implementations for yet more regular expression syntaxes. – Stéphane Chazelas Apr 24 '18 at 10:18
  • @StéphaneChazelas thank for the links, I try to use EREs most of the time. I can't stand the synthax of BREs – Kiwy Apr 24 '18 at 10:32
  • Can you look at this please: https://stackoverflow.com/questions/56459572/replace-a-string-in-a-file-with-contents-copied-from-another-file – devdoe Jun 05 '19 at 12:34
  • i used sed 's/something-with-variable/'something-with"$varaible"-something-else''... meaning, if i used single quote for the sed, then the replacement string i had to use single quote and inside use double quote – soMuchToLearnAndShare Jul 21 '20 at 07:43
33

This worked:

cat update_via_sed.sh | sed 's/'"$old_run"'/'"$new_run"'/'

As I want to 'reuse' the same file I actually use this for anyone wishing a similar approach:

cat update_via_sed.sh | sed 's/'"$old_run"'/'"$new_run"'/' > new_update; mv -f new_update update_via_sed.sh

The above created a new file then deletes the current file than rename the new file to be the current file.

Michael Durrant
  • 42,013
  • 5
    You don't need cat, mv, or extra '' unless you're going to have special characters in them. So it's sed -i s/"$old"/"$new"/ update_via_sed.sh. However be aware that this does not work for any value of old and new. e.g. it must not contain / etc., as $old is still a search and $new a replacement pattern where some characters have special meanings. – frostschutz Mar 25 '13 at 16:25
  • 3
    sed -i "s/$old/$new/" is ok too (with the above precautions). sed usually considers the separator to be the first character after the s command - i.e. if your variables contain slashes / but not lets say at (@), you can use "s@$old@$new@". – peterph Mar 25 '13 at 17:23
23

You can use variables in sed as long as it's in a double quote (not a single quote):

sed "s/$var/r_str/g" file_name >new_file

If you have a forward slash (/) in the variable then use different separator like below

sed "s|$var|r_str|g" file_name >new_file
EightBitTony
  • 21,373
mani
  • 231
  • 1
    +1 for mentioning needing to use double quotes. That was my problem. – speckledcarp Nov 16 '16 at 17:01
  • 1
    Exactly what I was looking for, including the use of | as in my case, the variables contains a path so it has / in it – yorch Nov 17 '16 at 21:51
  • For some reasons the pipe | worked for me on MacOS 10.14, but other seperators like @ or # not. I had forward slashes in my env var too. – davidenke Oct 22 '18 at 12:30
21

'in-place' sed (usng the -i flag) was the answer. Thanks to peterph.

sed -i "s@$old@$new@" file
Michael Durrant
  • 42,013
  • 2
    You have the quotes in the wrong place. quotes must be around variables, not s@ (which is not special to the shell in any way). Best is to put everything inside quotes like sed -i "s@$old@$new@" file. Note that -i is not a standard sed option. – Stéphane Chazelas Mar 26 '13 at 14:20
  • how can I escape $ in this command. Like, I am going to change $xxx as a whole to the value of a variable $JAVA_HOME. I tried sed -i "s@\$xxx@$JAVA_HOME@" file, but error says no previous regular expression – hakunami Nov 06 '14 at 08:07
3

man bash gives this about single quoting

Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

Whatever you type on the command line, bash interprets it and then it sends the result to the program it is supposed to be sent to.In this case, if you use sed 's/$old_run/$new_run/', bash first sees the sed, it recognises it as an executable present in $PATH variable. The sed executable requires an input. Bash looks for the input and finds 's/$old_run/$new_run/'. Single quotes say bash not to interpret the content in them and pass them as they are. So, bash then passes them to sed. Sed gives an error because $ can occur only at the end of line.

Instead if we use double quotes, i.e., "s/$old_run/$new_run/", then bash sees this and interprets $old_run as a variable name and makes a substitution (this phase is called variable expansion). This is indeed what we required.

But, you have to be careful using double quotes because, they are interpreted first by bash and then given to sed. So, some symbols like ` must be escaped before using them.

nitishch
  • 461
2
$ echo $d1 | sed -i 's/zkserver1/'${d1}'/g' deva_file
sed: -e expression #1, char 26: unterminated `s' command

In $d1 I am storing values

I cannot replace the value from the variable, so I tried the following command it is working:

echo $d1 | sed -i 's/zkserver1/'"${d1}"'/g' deva_file

missing double quotation mark in "${d1}", that was the error.

GAD3R
  • 66,769
deva
  • 21
1

At risk of being flippant - have you considered perl.

Which - amongst other things - is quite good at doing 'sed style' operations, but also a more fully featured programming thingummy.

It looks like in your example, what you're actually trying to do is increment a number.

So how about:

#!/usr/bin/env perl
use strict;
use warnings 'all'; 

while ( <DATA> ) {
    s/old_name_(\d+)/"old_name_".($1+1)/e;
    print;
}

__DATA__
old_name_932

or as a one liner - sed style:

perl -pe 's/old_name_(\d+)/"old_name_".($1+1)/e'
Sobrique
  • 4,424
-3

break the string

bad => linux see a string $old_run :

sed 's/$old_run/$new_run/'

good => linux see a variable $old_run:

sed 's/'$old_run'/'$new_run'/'
marcdahan
  • 97
-4

You can also use Perl:

perl -pi -e ""s/$val1/$val2/g"" file
Kusalananda
  • 333,661
Samba
  • 1
  • 1
    Consider variables that may have spaces in their values. The empty strings around the Perl expression ("") won't do anything. – Kusalananda Jun 22 '18 at 07:25