How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

Question

CentOS 6.3

I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting).

The template file contains the following:

Alert Report

Alert has triggered at TMPDATE

HEADERS
-------
TMPHEADERS

SOURCE IP
---------
TMPSOURCEIP

I want my script to find and replace the "TMP" place holders with actual information relevant to the alert.

The date and source IP variables seem to work fine but if/when I try to find/replace the TMPHEADERS, it causes strange symptoms including the entire output to be blank. I'm reasonably certain that it has to do with the headers containing lots of special characters (@ % & etc).

Can someone please advise on how best to accomplish this via bash?

My current code looks like this:

cat /test/emailtemplate | sed s/TMPHEADERS/"$HEADERS"/ > /test/output

UPDATE

As per request, here's an example set of headers I'm trying to insert into the template file (IP addresses and email address have been changed to protect the innocent :-) ):

From foo@yahoo.com Thu Apr 25 20:18:19 2013
Return-Path: <foo@yahoo.com>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
for <test@example.com>; Thu, 25 Apr 2013 20:18:19 -0700
Received: from [108.108.108.108] by nm30.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:19 -0000
Received: from [98.138.87.11] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 92334348.43667.bm@omp10122422401.mail.ne1.yahoo.com
Received: (qmail 29701 invoked by uid 65501); 26 Apr 2013 03:18:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1366946297; bh=yjMzVONHAyJxQob7tLNdIN2RpfGHWVw1Kb5Qr+enYF8=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=CQGilBx8NertE4j75dsfsdfs+IKRzIx5vlH5YdzqFLd4ThfEyMs11sdfsdfsdXsAH1yteACnwSER+QXJQ80BfLGbJnIWm+29I8A4geOPIHGKoOUCnPaD+/0bHfAps0JIcwEju8Tcvg4VDVWw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
b=5TIRL55VM2J2lPLsX9iCE4sdlkfjlEM2245M6qzg1oGrnZd61ykL4xQSsc3cYNz95fwNy67aRRC89n6xcti28ee5rjmlK0MDIskSB5sKlv165mNjmzF1LNx6uFXalI8QGSwiQt2uWLYvI7RrTVeZFELDfFVZyqygEl5k=;
X-YMail-OSG: EDT4ym4VM1lVQMdtAQ5zqfE59jR1Mtip4vVL1fBzNxFdGvA
YjAJ3MXC1EusloknrsPx3drxzR1b4PFErK.UhdgWePhK7TTCHhhju4XP4i7x
76WASceqp77T5itvZmilv5UuICJw3BCEd0fdADctfBYhLNwoALxjp6cnJMmE
Z4dYVtlp5vUFqg1pHxqGOXqrtjeZffM4dMftnn.Q8LlVEkj3pZ6ZJV_kKFtj
vGEGS5PAW0tIHWPEqVERYzmDOfF5sVSQLayPi6EM_i1OE038434laijEWbH0
nZt1Vkg3syO0t1BaTLN4B1bXeS8cv3GlbLO4ot7zVwA3sH4UhsC5M6xiWNFU
3iroObJ5BObL99VO3ktvC4KzekAWJ_fE85TQJhQKj6Iolgb4xlWa2x414xuA
awO4pJI9grDjycUcmhmKwLZEt_.0OBLfSi5MSviaiCNMuU5qIdHm7VCGdORP
Mc68rDkpmJE9I9Z.QZfhH5cFxqqmpyIOMTs0iIBGYz5d9QHMWCuo-
Received: from [102.102.102.102] by web12341105.mail.ne1.yahoo.com via HTTP; Thu, 25 Apr 2013 20:18:17 PDT
X-Rocket-MIMEInfo: 002.001,VGVzdGluZyAxMjMgTGEgbGEgbGEBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.141.536
Message-ID: <136634397.44849.YahooMailNeo@web126205.mail.ne1.yahoo.com>
Date: Thu, 25 Apr 2013 20:18:17 -0700 (PDT)
From: Test Account <foo@yahoo.com>
Reply-To: Test Account <foo@yahoo.com>
Subject: Test
To: "test@example.com" <test@example.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="102743210-1541163991-1366946297=:49149"

Simple text manipulation tools have limitations, they are suitable only for basic template handling. Better see Tool to create text files from a template. Has links to similar [SE] and [SF] questions, see those too. — manatwork, Apr 28 '13 at 16:31
@terdon Absolutely. I updated the question to include some example headers (with different ips and email addresses). — Mike B, Apr 28 '13 at 16:44

score 8 · Answer 1 · edited Sep 25 '14 at 17:18

This is difficult in sed because in sed s/TMPHEADERS/"$HEADERS"/, the value of the variable is substituted by the shell before the command reaches sed, so characters like &\/ that appear in $HEADERS have a special meaning in the replacement text. Sed does not have a variable lookup facility.

This task is fairly easy with awk. Pass the HEADERS variable in the environment:

export HEADERS
</test/emailtemplate awk '
    {gsub(/^TMPHEADERS$/, environ["HEADERS"]); print}
' > /test/output

or as an awk variable:

</test/emailtemplate awk -v HEADERS="$HEADERS" '
    {gsub(/^TMPHEADERS$/, HEADERS); print}
' > /test/output

+1 to this approach, configuring awk's environment is much less error-prone than trying to wrestle with sed's syntax. — dimo414, Mar 09 '21 at 22:12

score 3 · Answer 2 · edited Apr 28 '13 at 20:10

3

There are several possibilities:

sed You would have to escape every char in "$HEADERS" to make the sed call safe.

HEADERS_ESCAPED="$(echo "$HEADERS" | sed -n 's/./\\&/gp')"
# double-useless use of cat BTW
sed "s/TMPHEADERS/$HEADERS_ESCAPED/" /test/emailtemplate > /test/output

BTW: There is absolutely no reason to call sed once for every change:

sed -n -e s/X/"$X"/ -e s/Y/"$Y" -e s/Z/"$Z"/ -e p inputfile > outputfile

bash itself

while IFS= read -r line; do
  line="${line/TMPDATE/"$TMPDATE"}"
  # ...
  line="${line/TMPHEADERS/"$HEADERS"}"
  printf '%s\n' "$line"
done <inputfile >outputfile

edited Apr 28 '13 at 20:10

Gilles 'SO- stop being evil'

829,060

answered Apr 28 '13 at 16:56

Hauke Laging

90,279

@terdon Indeed, that was the wrong variable. sed does not expand anything (due to the escaping of every char in the variable). What do you mean? Have you tried this, doesn't it work? – Hauke Laging Apr 28 '13 at 17:08
My bad, I always quote my sed commands (I thought that was necessary) and bash variables are not expanded (I mean that 's/foo/$bar/' actually printed $bar and not the variable's contents). Your command worked fine without the quotes (as you had posted it) it failed when I put it in single quotes. Sorry, force of habit, feell free to roll back my edit. – terdon Apr 28 '13 at 17:09
Thanks for the prompt help but I'm getting an error: sed: -e expression #1, char 121: unterminateds' commandI'm guessing I must have configured the command wrong:sed "s/TMPHEADERS/$HEADERS_ESCAPED/" /home/foo/scripts/fooalert/emailtemplate > /home/foo/scripts/fooalert/testing1234` I've confirmed that the headers are now all escaped properly. – Mike B Apr 29 '13 at 00:45
@MikeB Can you find out what char causes the problem? I am not sure whether it is possible to have a newline in the replacement text. Is it possible that $HEADERS_ESCAPED contains a newline? – Hauke Laging Apr 29 '13 at 01:00

Kusalananda · Answer 3 · 2017-09-25T16:55:01.610

Using the m4 macro processor to replace TMPSOURCEIP and TMPDATE with things defined on the command line and TMPHEADERS with whatever is in the file headers.txt. The template is in template.txt.

$ m4 -DTMPDATE="$(date)" -DTMPSOURCEIP="1.1.1.1" -DTMPHEADERS='include(headers.txt)' template.txt
Alert Report

Alert has triggered at Mon Sep 25 18:46:34 CEST 2017

HEADERS
-------
From foo@yahoo.com Thu Apr 25 20:18:19 2013
Return-Path: <foo@yahoo.com>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
(etc.)


SOURCE IP
---------
1.1.1.1

If you already have the headers in a variable, you may obviously replace -DTMPHEADERS='include(headers.txt)' with -DTMPHEADERS="$headers".

score 1 · Answer 4 · edited Sep 25 '14 at 17:20

I'm not sure I understand what you want to do. If you want to, for example, replace the string TMPHEADERS with the long list of header info you posted, this Perl solution can do it:

perl -e 'open(A,"header"); ## open the header file
         $h=join("",<A>); ## save its contents in a string
         while(<>){ ## go through the input file
            s/TMPHEADERS/$h/; ## replace
            print        ## print each line
         }' template

The file template is the template from your question and the file header has the header lines you posted. The result is

Alert Report

Alert has triggered at TMPDATE

HEADERS
-------
From foo@yahoo.com Thu Apr 25 20:18:19 2013
Return-Path: <foo@yahoo.com>
Received: from nm30-vm0.bullet.mail.ne1.yahoo.com (nm30-vm0.bullet.mail.ne1.yahoo.com [98.138.11.36])
by serv.example.com (8.14.4/8.14.4) with ESMTP id r3Q3IJVV009411
for <test@example.com>; Thu, 25 Apr 2013 20:18:19 -0700
Received: from [108.108.108.108] by nm30.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:19 -0000
Received: from [98.138.87.11] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 26 Apr 2013 03:18:17 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 92334348.43667.bm@omp10122422401.mail.ne1.yahoo.com
Received: (qmail 29701 invoked by uid 65501); 26 Apr 2013 03:18:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1366946297; bh=yjMzVONHAyJxQob7tLNdIN2RpfGHWVw1Kb5Qr+enYF8=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=CQGilBx8NertE4j75dsfsdfs+IKRzIx5vlH5YdzqFLd4ThfEyMs11sdfsdfsdXsAH1yteACnwSER+QXJQ80BfLGbJnIWm+29I8A4geOPIHGKoOUCnPaD+/0bHfAps0JIcwEju8Tcvg4VDVWw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
s=s1024; d=yahoo.com;
h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
b=5TIRL55VM2J2lPLsX9iCE4sdlkfjlEM2245M6qzg1oGrnZd61ykL4xQSsc3cYNz95fwNy67aRRC89n6xcti28ee5rjmlK0MDIskSB5sKlv165mNjmzF1LNx6uFXalI8QGSwiQt2uWLYvI7RrTVeZFELDfFVZyqygEl5k=;
X-YMail-OSG: EDT4ym4VM1lVQMdtAQ5zqfE59jR1Mtip4vVL1fBzNxFdGvA
YjAJ3MXC1EusloknrsPx3drxzR1b4PFErK.UhdgWePhK7TTCHhhju4XP4i7x
76WASceqp77T5itvZmilv5UuICJw3BCEd0fdADctfBYhLNwoALxjp6cnJMmE
Z4dYVtlp5vUFqg1pHxqGOXqrtjeZffM4dMftnn.Q8LlVEkj3pZ6ZJV_kKFtj
vGEGS5PAW0tIHWPEqVERYzmDOfF5sVSQLayPi6EM_i1OE038434laijEWbH0
nZt1Vkg3syO0t1BaTLN4B1bXeS8cv3GlbLO4ot7zVwA3sH4UhsC5M6xiWNFU
3iroObJ5BObL99VO3ktvC4KzekAWJ_fE85TQJhQKj6Iolgb4xlWa2x414xuA
awO4pJI9grDjycUcmhmKwLZEt_.0OBLfSi5MSviaiCNMuU5qIdHm7VCGdORP
Mc68rDkpmJE9I9Z.QZfhH5cFxqqmpyIOMTs0iIBGYz5d9QHMWCuo-
Received: from [102.102.102.102] by web12341105.mail.ne1.yahoo.com via HTTP; Thu, 25 Apr 2013 20:18:17 PDT
X-Rocket-MIMEInfo: 002.001,VGVzdGluZyAxMjMgTGEgbGEgbGEBMAEBAQE-
X-Mailer: YahooMailWebService/0.8.141.536
Message-ID: <136634397.44849.YahooMailNeo@web126205.mail.ne1.yahoo.com>
Date: Thu, 25 Apr 2013 20:18:17 -0700 (PDT)
From: Test Account <foo@yahoo.com>
Reply-To: Test Account <foo@yahoo.com>
Subject: Test
To: "test@example.com" <test@example.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="102743210-1541163991-1366946297=:49149"


SOURCE IP
---------
TMPSOURCEIP

You can extend this for two sets of headers as follows:

perl -e 'open(A,"header1");
         $h1=join("",<A>);
         open(B,"header2");
         $h2=join("",<B>);
         while(<>){
            s/TMPHEADERS/$h1/;
            s/TMPSOURCEIP/$h2/;
            print        
         }' template

I respect this answer but unfortunately I don't know perl well enough to maintain it on an on-going basis. I realize that's not a good answer (or reason not to use more efficient code). Thanks. — Mike B, Apr 29 '13 at 00:48

How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

UPDATE

4 Answers4

Linked