How do I bring HEREDOC text into a shell script variable?

Question

I'm trying to bring HEREDOC text into a shell script variable in a POSIX compliant way. I tried like so:

#!/bin/sh

NEWLINE="
"

read_heredoc2() {
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    echo "${read_heredoc_line}"
  done
}

read_heredoc2_result="$(read_heredoc2 <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC
)"

echo "${read_heredoc2_result}"

That produced the following which is wrong:

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _  | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

The following works but I don't like how clunky it is by using a random output variable:

#!/bin/sh

NEWLINE="
"

read_heredoc1() {
  read_heredoc_first=1
  read_heredoc_result=""
  while IFS="$NEWLINE" read -r read_heredoc_line; do
    if [ ${read_heredoc_first} -eq 1 ]; then
      read_heredoc_result="${read_heredoc_line}"
      read_heredoc_first=0
    else
      read_heredoc_result="${read_heredoc_result}${NEWLINE}${read_heredoc_line}"
    fi
  done
}

read_heredoc1 <<'HEREDOC'

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|                                                



HEREDOC

echo "${read_heredoc_result}"

Correct output:

                        _                            _ _            
                       | |                          | (_)           
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___ 
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |                                                
            |___/|_|

Any ideas?

If the banner is used only once, use cat with a here-document directly. If it's used in many places in the script, store it on file an cat it from there, just like /etc/motd is used on some systems. — Kusalananda, Jan 28 '17 at 07:42
Note that the problem you're having is actually a Bash bug - you had a POSIX solution already in your original attempt, which will work fine in ksh, dash, ash, and the oldest Bourne shell I can find. Bash command substitution parsing is weird, and used to be even more broken. — Michael Homer, Jan 29 '17 at 04:02
@MichaelHomer Oh, interesting! Wow, I'm on the latest Fedora 25, bash 4.3.43-4.fc25 — Kevin, Jan 29 '17 at 04:08
Yeah. If you can get a 3-series version to try the script with, it breaks even sooner at the first backtick, so it's improving. I'm not sure whether they consider it a bug - arguably POSIX doesn't explicitly prohibit this behaviour, but it's fairly clear that command substitution contains all the characters until the ) and that quoted heredocs don't have expansions. — Michael Homer, Jan 29 '17 at 04:13
I've asked a question to try to sort out what POSIX really requires in this situation. — Michael Homer, Jan 29 '17 at 05:21
@Kevin I have updated my answer (at the start) perhaps it will be useful. — , Feb 23 '17 at 22:22

score 18 · Accepted Answer · edited Feb 23 '17 at 17:34

The problem is that, in Bash, inside $( ... ) escape (and other) sequences get parsed, even though the heredoc itself wouldn't have them. You get a doubled line because \ escapes the line break. What you're seeing is really a parsing issue in Bash - other shells don't do this. Backticks can also be a problem in older versions. I have confirmed that this is a bug in Bash, and it will be fixed in future versions.

You can at least simplify your function drastically:

func() {
    res=$(cat)
}
func <<'HEREDOC'
...
HEREDOC

If you want to choose the output variable it can be parameterised:

func() {
    eval "$1"'=$(cat)'
}
func res<<'HEREDOC'
...
HEREDOC

Or a fairly ugly one without eval:

{ res=$(cat) ; } <<'HEREDOC'
...
HEREDOC

The {} are needed, rather than (), so that the variable remains available afterwards.

Depending on how often you'll do this, and to what end, you might prefer one or another of these options. The last one is the most concise for a one-off.

If you're able to use zsh, your original command substitution + heredoc will work as-is, but you can also collapse all of this down further:

x=$(<<'EOT'
...
EOT
)

Bash doesn't support this and I don't think any other shell that would experience the problem you're having does either.

One issue I'm finding is that $(cat) is stripping trailing newlines — Kevin, Jan 28 '17 at 08:47

score 7 · Answer 2 · 2017-02-23T22:20:26.833

About the OP solution:

You do not need an eval to assign a variable if you allow some constant variable to be used.
the general structure of calling a function that receives the HEREDOC could also be implemented.

A solution that works in all (reasonable) shells with both items solved is this:

#!/bin/bash
nl="
"

read_heredoc(){
    var=""
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done 
}


read_heredoc <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|

HEREDOC

read_heredoc2_result="$str"

printf '%s' "${read_heredoc2_result}"

A solution for the original question.

A solution that works since bash 2.04 (and recent zsh, lksh, mksh).
Look below for a more portable (POSIX) version.

#!/bin/bash
read_heredoc() {
    IFS='' read -d '' -r var <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|



HEREDOC

}

read_heredoc
echo "$var"

The core command

IFS='' read -d '' -r var <<'HEREDOC'

works as follows:

The word HEREDOC is (single) quoted to avoid any expansion of the text that follows.
The "here doc" contents are served in the stdin with <<.
The option -d '' forces read to slurp the whole content of the "here doc".
The -r option avoids interpretation of backslash quoted characters.
The core command is similar to read var.
And the last detail is IFS='', which will avoid that read remove leading or trailing characters in the default IFS: spacetabnewline.

In ksh, the null value for the -d '' option doesn't work.
As a workaround, if the text has no "carriage return", a -d $'\r' works (if a $'\r' is added to the end of each line, of course).

An added (in comments) requirement is to generate a POSIX compliant solution.

POSIX

Extending the idea to make it run only with POSIX options.
That means mainly no -d for read. That forces a read for each line.
That, in turn forces the need to capture a line at a time.
Then, to build the var a trailing new line must be added (as the read removed it).

#!/bin/sh

nl='
'

read_heredoc() {
    unset var
    while IFS="$nl" read -r line; do
        var="$var$line$nl"
    done <<\HEREDOC

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \ 
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/ 
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___| 
             __/ | | 
            |___/|_| 



HEREDOC

}

read_heredoc
printf '%s' "$var"

That works (and has been tested) in all reasonable shells.

The HEREDOC embedded in the POSIX labelled version seems like a typo. Why would it make any sense to embed the here document in the function? I tested a modified copy on the POSIX version that did not not embed the here document but assigned the variable more like the other examples did, and the update works on a very old Bourne shell from SCO OpenServer 5.0.7. I don't see any reason for the \HEREDOC, but maybe there is more to this that I do not understand. — kbulgrien, Feb 27 '23 at 23:31

Kevin · Answer 3 · 2017-01-29T07:56:18.213

3

To support trailing newlines, I combined the answer from @MichaelHomer and my original solution. I didn't use the suggested workarounds from the link that @EliahKagan noted because the first one uses magical strings and the last two weren't POSIX compliant.

#!/bin/sh

NEWLINE="
"

read_heredoc() {
  read_heredoc_result=""
  while IFS="${NEWLINE}" read -r read_heredoc_line; do
    read_heredoc_result="${read_heredoc_result}${read_heredoc_line}${NEWLINE}"
  done
  eval $1'=${read_heredoc_result}'
}

read_heredoc heredoc_str <<'HEREDOC'

                        _                            _ _
                       | |                          | (_)
  _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | |_ _ __   ___
 | '_ ` _ \| | | | '_ \| |/ _` |/ __/ _ \/ _ \| '_ \| | | '_ \ / _ \
 | | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
 |_| |_| |_|\__, | .__/|_|\__,_|\___\___|\___/|_| |_|_|_|_| |_|\___|
             __/ | |
            |___/|_|




HEREDOC

echo "${heredoc_str}"

edited Jan 29 '17 at 07:56

answered Jan 29 '17 at 04:44

Kevin

411

@sorontar I just tested it and the trailing new line didn't seem erased to me. I'm not sure what you mean by the first point about using a variable to read the first line. Regarding the eval, it's only used for the name of the "output" variable. If we assume trusted users of the function, are there any other issues using eval in this example? – Kevin Jan 29 '17 at 06:35
@sorontar Interesting point about the trailing newline. A here-document must end with a newline, followed by the delimiter and a newline, but it doesn't say whether the newline before the delimeter should be part of the string or not: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 – Kevin Jan 29 '17 at 06:53
1

Yes, A here-document must end with a newline, and you code remove it. Every "text line must end on a newline". Search in definitions of text file. – Jan 29 '17 at 07:04
3.195 Incomplete Line A sequence of one or more non- characters at the end of the file. – Jan 29 '17 at 07:07
3.206 Line A sequence of zero or more non- characters plus a terminating character. – Jan 29 '17 at 07:08
@sorontar That makes sense, thanks for the references! – Kevin Jan 29 '17 at 07:21
Use of 'read -r' breaks compatibility with some very old systems (i.e. SCO OpenServer 5.0.7). On that system, I had to use a slightly modified copy of the POSIX variant posted by @user232326 (I moved the here document outside the function). – kbulgrien Feb 27 '23 at 23:36

ctx · Answer 4 · 2017-02-23T17:11:03.593

2

Useless use of cat (quote \ and `):

myplaceonline="
                       _                            _ _            
 _ __ ___  _   _ _ __ | | __ _  ___ ___  ___  _ __ | (_)_ __   ___ 
| '_ \` _ \\| | | | '_ \\| |/ _\` |/ __/ _ \\/ _ \\| '_ \\| | | '_ \\ / _ \\
| | | | | | |_| | |_) | | (_| | (_|  __/ (_) | | | | | | | | |  __/
|_| |_| |_|\\__, | .__/|_|\\__,_|\\___\\___|\\___/|_| |_|_|_|_| |_|\\___|
       |___/|_
"

Or without quoting:

myplaceonline="$(figlet myplaceonline)"

edited Feb 23 '17 at 17:11

answered Feb 23 '17 at 16:10

ctx

2,495

Latter is not POSIX. – phk Feb 23 '17 at 16:40
you are right, now it should be? – ctx Feb 23 '17 at 17:13
1

@ctx Thanks for the answer, but an unstated requirement of mine is that I want to be able to bring in unquoted HEREDOCs (partly because I'm making this into a public API function for a library called posixcube). The accepted answer is ultimately correct that there is a Bash bug processing nested HEREDOCs in command substitution (see the comment from Michael Homer in the question which links to another question which links to the Bash mailing list where the bug is confirmed by a Bash maintainer). My answer above on Jan 29th is my workaround solution and works well. – Kevin Feb 23 '17 at 18:43

How do I bring HEREDOC text into a shell script variable?

4 Answers4

A solution for the original question.

POSIX

Linked