Iterate through multiline string line by line

Question

I would like to process a multiline string and iterate it line by line, in a POSIX shell (/bin/sh) on a BSD platform. Bash is not included in the base BSD-distribution and has a GPL license - so I am trying to make it universally work with /bin/sh instead.

I found a solution using a pipe, however in the regular /bin/sh shell, these a processed in a separate process, meaning the following does not work:

MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="original value"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 SOMEVAR="updated value"
 echo "this is a single line: ${SINGLELINE}"
 echo "SOMEVAR is now: ${SOMEVAR}"
done

echo "Final SOMEVAR is unchanged: ${SOMEVAR}"

In the above example, it accomplishes what I want, except for the fact that changes to variables such as ${SOMEVAR} are not accessible outside the while loop.

My question: how can I accomplish something like the above without this restriction? Note that many solutions require Bash, whereas I am using the standard POSIX-shell /bin/sh.

I can not reproduce your issue using /bin/sh on a BSD system. There is nothing in this code that would require bash. — Kusalananda, Mar 31 '19 at 10:57
My apologies. In trying to create a minimized sample of the actual issue, i made some errors. I updated the code sample now, and it should reflect the problem correctly now. The example indeed does not require Bash. — Steiner, Mar 31 '19 at 11:09
There is no Bourne shell on the BSDs. Nor is the Bourne shell standard. Do not conflate the Bourne shell with a POSIX-conformant sh. — JdeBP, Mar 31 '19 at 11:29

Stéphane Chazelas · Accepted Answer · 2019-03-31T18:03:43.970

You could use a here document:

while IFS= read -r SINGLELINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done << EOF
$MULTILINE
EOF
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

Depending on the sh implementation, here-documents are implemented either as a deleted temporary file where the shell has stored the expansion of the variable followed by newline beforehand, or a pipe to which the shell feeds the expansion of the variable followed by newline. But in either case, except in the original Bourne shell (a shell that is no longer in use these days and is not a POSIX compliant shell), the command being redirected is not run in a subshell (as POSIX requires).

or you could use split+glob:

IFS='
' # split on newline only
set -o noglob
for SINGLELINE in $MULTILINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

But beware it skips empty lines.

Both solutions worked well for me with /bin/sh. I think this solution is better than the one Kusalananda provided, though his answer is very informative too. I particularly like the for-loop solution since it probably is faster because it does not use read which is very slow. Thanks! — Steiner, Mar 31 '19 at 15:54
@Steiner (Just had to quickly double check so that I didn't have a for-loop solution, which I fortunately did not have) — Kusalananda, Mar 31 '19 at 18:41

Kusalananda · Answer 2 · 2019-03-31T11:35:55.510

You would read directly from the file without the pipeline. This avoids running the while loop in a subshell, which allows you to see the changed value of $SOMEVALUE after the loop.

SOMEVAR="original value"

while IFS= read -r SINGLELINE
do
    SOMEVAR="updated value"
    printf 'this is a single line: %s\n' "$SINGLELINE"
    printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
done <"$SOMEFILE"

printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"

If you insist on having your $MULTILINE variable, then write that to a file and read it from there:

tmpfile=$(mktemp)
printf '%s\n' "$MULTILINE" >"$tmpfile"

while ...; do
   ...
done <"$tmpfile"
rm "$tmpfile"

Also related:

Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

An answer to the above linked question also suggests writing your program in such a way that all uses of $SOMEVAR occurs within the subshell at the end of the pipeline:

MULTILINE=$(cat "$SOMEFILE")
SOMEVAR="original value"

printf '%s\n' "$MULTILINE" | {
    while IFS= read -r SINGLELINE
    do
        SOMEVAR="updated value"
        printf 'this is a single line: %s\n' "$SINGLELINE"
        printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
    done

    printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"
}

Also possibly related:

Why is using a shell loop to process text considered bad practice?

Other questions that may be of interest:

Indeed this works, but requires a file to be fed to the while-loop. My actual code is more complex, and requires a modified multiline string to be fed to the while-loop. Your solution to write this multiline string to a temporary file works, but is there a solution that does not require writing a temporary file? — Steiner, Mar 31 '19 at 11:28
@Steiner See update answer. There's not much else you could do without using features of specific shells. — Kusalananda, Mar 31 '19 at 11:33
Thank you that answers my question. The suggestion at the end and the links provided are very helpful too! — Steiner, Mar 31 '19 at 11:43
@Steiner ... except for using a here-document. Well, there's a thing I didn't think of :-) — Kusalananda, Mar 31 '19 at 18:42

score 0 · Answer 3 · answered Mar 31 '19 at 11:11

It works for me :

$ cat bin/test
#! /bin/sh
SOMEFILE=$1
MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="blah"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 echo "this is a single line: ${SINGLELINE}"
 echo "but accessing this var fails: ${SOMEVAR}"
done

and

$ bin/test bin/test
this is a single line: #! /bin/sh
but accessing this var fails: blah
this is a single line: SOMEFILE=$1
but accessing this var fails: blah
this is a single line: MULTILINE="`cat ${SOMEFILE}`"
but accessing this var fails: blah
this is a single line: SOMEVAR="blah"
but accessing this var fails: blah
this is a single line:
but accessing this var fails: blah
this is a single line: echo "${MULTILINE}" | while IFS= read -r SINGLELINE
but accessing this var fails: blah
this is a single line: do
but accessing this var fails: blah
this is a single line:  echo "this is a single line: ${SINGLELINE}"
but accessing this var fails: blah
this is a single line:  echo "but accessing this var fails: ${SOMEVAR}"
but accessing this var fails: blah
this is a single line: done
but accessing this var fails: blah

My apologies. In trying to create a minimized sample of the actual issue, i made some errors. I updated the code sample now, and it should reflect the problem correctly now. — Steiner, Mar 31 '19 at 11:13

Iterate through multiline string line by line

3 Answers3