How do I process the whole file in one buffer in sed, without GNU -z option?

Question

For some problems like matching a pattern over an unknown number of lines or "replace the last occurence of ..." the option -z of GNU sed is really helpful. How can I achieve the same thing portable?

Example: I have a file

yellow, green,
blue, black, purple,
orange,
white, red, brown
are some colours

and I want to replace the last comma of the file with and. Note that it is unknown in which line or where in that line the comma is. With GNU sed I can do

sed -z 's/\(.*\),/ \1 and/'

to get the desired output

yellow, green,
blue, black, purple,
orange,
white, red and brown
are some colours

How can I do it in a portable way, that will run with any POSIX sed?

Answer 1

In pure POSIX sed you have to paste all lines by yourself. While some people do this with N inside a loop, the easiest approach is to append to the hold space with the H;1h;$!d;x pattern:

• H appends each line to the hold space. Unfortunally, appending the first line will add a newline to the beginning of the buffer, so
• 1h will override the hold space for the first line to avoid the wrong newline.
• $!d will end processing for all lines exept for the last one. They don't need to be printed, because they are stored in the hold space
• x will be executed only after the last line (for all other lines the d did stop further command processing) and it will exchange hold space and pattern space, so after this command the whole file that was collected in the hold space will be in pattern space, just like it would be with the -z option of GNU sed. Of course you could also use g instead of x, but this will produce a lot of copying, so x is faster.

So the script for the example will look like:

sed 'H;1h;$!d;x;s/\(.*\),/\1 and/'

Please note that processing a file like this is not a good idea for very large files, because this will use lots of RAM.

Answer 2

sed is for doing simple s/old/new on individual strings, that is all. Almost any time you find yourself using constructs other than s, g, and p (with -n) and certainly any time you find yourself talking about "hold space" you are using the wrong tool. For anything more complicated than s/old/new, like this task, you should just use awk instead. The following will work using any awk in any shell on any UNIX box, doesn't store the whole file in memory, and is trivial to tweak if/when you want to additionally do anything else to the text:

$ cat tst.awk
/,/ { printf "%s", prev; prev="" }
{ prev = prev $0 ORS }
END {
    if ( match(prev,/.*,/) ) {
        prev = substr(prev,1,RLENGTH-1) " and" substr(prev,RLENGTH+1)
    }
    printf "%s", prev
}

$ awk -f tst.awk file
yellow, green,
blue, black, purple,
orange,
white, red and brown
are some colours

You COULD do the job more briefly in awk by slurping the whole file into memory and writing this cryptic rune:

$ awk '{r=r$0 ORS} END{h=r;sub(/,[^,]+$/,"",h);sub(/.*,/,"",r);printf "%s and%s",h,r}' file
yellow, green,
blue, black, purple,
orange,
white, red and brown
are some colours

but the point is that, unlike with sed, you don't have to.