sed/awk remove newline on two pattern matches

Question

I have a .pem file that after some formatting it shows like this:

-----BEGIN
RSA
PRIVATE
KEY-----
MIIEogIBAAKCAQEAoK3D4mMIRnzPaYqHidgpBnDDzLlcYYd0GoB0pQGyGSHDW7KO3K+VeJP90GhE
ZTEWJLp2N5DR/KT+5Vg7cgdx/GCCrnlbW0McP/IvkYAuWCgbzoXH9eE+kDRtAmurBYCk7OTOwQ26
..........................LONG LONG LONG KEY................................
QRRLFsXua9spUh0yPd163IZStKZMhZBPJfMaqbi1WF+j21DdYyS0qpaZQRjFzRLvGuXeGLZG/COD
ip10XRJMMXY0m14bqOTcqKTya/5PTJIjDWC22+soIjLy0ZjWKo9n05Oal2t4q35kbos=
-----END
RSA
PRIVATE
KEY-----

And I obviously need to be like:

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAoK3D4mMIRnzPaYqHidgpBnDDzLlcYYd0GoB0pQGyGSHDW7KO3K+VeJP90GhE
ZTEWJLp2N5DR/KT+5Vg7cgdx/GCCrnlbW0McP/IvkYAuWCgbzoXH9eE+kDRtAmurBYCk7OTOwQ26
..........................LONG LONG LONG KEY................................
QRRLFsXua9spUh0yPd163IZStKZMhZBPJfMaqbi1WF+j21DdYyS0qpaZQRjFzRLvGuXeGLZG/COD
ip10XRJMMXY0m14bqOTcqKTya/5PTJIjDWC22+soIjLy0ZjWKo9n05Oal2t4q35kbos=
-----END RSA PRIVATE KEY-----

How can I do it using awk or sed?

All the tutorials that I've found online shows how to do it for the whole text or for one type of pattern like aaaabbbaaacccc which I found confusing.

I tried something like this:

tr ' ' '\n' < test.pem  | awk '/BEGIN\nRSA\nPRIVATE\n$/ { printf("%s\t", $0); next } 1' > test-format.pem

But that failed.

I don't know if it matters or not, but I'm doing this in a shell #!/bin/sh script in a python:3.5-alpine docker container.

Any help is greatly appreciated.

cuonglm · Accepted Answer · 2016-07-11T18:23:56.913

12

One way with sed:

sed -e '/^---/ {
  :1
  $!N
  s/\n/ /
  /---$/!b1
}' <file

This approach does the job straight way:

If line start with ---, then enter the code block
Inside code block, we create a loop label named 1
We read Next line of the input if current line is not the last line $
Replace newline \n with a space
If next line does end with ---, break ! the loop, otherwise restart the loop b1

edited Jul 11 '16 at 18:23

answered Jul 11 '16 at 17:53

cuonglm

153,898

Could you elaborate on what's going on in this sed please? – Fadi Jul 11 '16 at 18:07
Gived +1 for you. – Luciano Andress Martini Jul 11 '16 at 18:13
If I could +1 you I would.. This is the fastest and straight to the point solution.. Thank you! – Fadi Jul 11 '16 at 18:29
@Fadi It looks like you're at 15 rep, so you should be able to now. – clk Jul 11 '16 at 18:35

score 3 · Answer 2 · answered Jul 11 '16 at 18:41

3

Suppose the key lines should be collected in one long line too:

sed '
    :a
    N
    /\n-\|-\n/{
        P
        D
    }
    s/\n/ /
    ta
    ' file

answered Jul 11 '16 at 18:41

Costas

14,916

score 3 · Answer 3 · answered Jul 11 '16 at 20:12

Here's another sed:

sed '/^---/,/---$/{H;/^---/h;/---$/!d;x;s/\n/ /g;}' infile

this one uses a range and the hold buffer. It appends each line in that range to the hold buffer, overwriting it if it's the start of range; it deletes the line if it's not the end of range else it exchanges buffers and replaces all newlines with spaces.
And a different approach with ed:

ed -s infile <<\IN
?^---?,.-s/$/ /
?^---?,$j
1,/---$/-s/$/ /
1,/---$/j
w
q
IN

This one uses four ranges: first it searches backwards (ed starts at end of file) and it appends a space to end of each line in the first range and then joins the lines in the second range (which are actually lines in the first one plus one more line); it then does the same but starting from the beginning of the file.

The ed solution will edit the file in-place, replace w with ,p if you just want to print the result. — don_crissti, Jul 11 '16 at 20:23

score 2 · Answer 4 · edited Jul 11 '16 at 18:34

2

Doing the posix version as requested in the comment:

#!/bin/sh
jump=1

while IFS= read -r lin; do

  case $lin in 
    ----*)
        jump=0
    ;;
    *----)
        jump=1
    ;;
  esac

  if [ "$jump" = "1" ]; then 
        printf '%s\n' "$lin" 
  else
        printf '%s' "$lin "
  fi

done <"$1"

To call it:

./script file.txt >thedest.txt

edited Jul 11 '16 at 18:34

cuonglm

153,898

answered Jul 11 '16 at 17:55

Luciano Andress Martini

6,628

You may not want to use shell loop for processing text http://unix.stackexchange.com/q/169716/38906 – cuonglm Jul 11 '16 at 17:56
Ok and if you do not have cut, sed, awk but just echo and while, how do you solve this problem? – Luciano Andress Martini Jul 11 '16 at 17:57
1

Then you must use the shell, of course, but use it the right way. Also, cut, sed, awk or any other standard tools can be seen in most *nix, unlike bash, which is only often seen in GNU/Linux. – cuonglm Jul 11 '16 at 18:01
Will this work for #!/bin/sh ? Also, the reason why I was asking for sed/awk is because I'm creating a new pem file and echoing the contents then I'm formatting it. Not sure how it will work with this method. – Fadi Jul 11 '16 at 18:06
I tested in my debian with ksh and dash and it works. I think it can run, but maybe you will need to replace the condition jump = 1 to a traditional if. – Luciano Andress Martini Jul 11 '16 at 18:09
Your one is fine, it will run in any POSIX shell. But you need while IFS= read -r, see the link I posted above – cuonglm Jul 11 '16 at 18:10
Thank you. You can add this to the post, i will allow you. – Luciano Andress Martini Jul 11 '16 at 18:11
This isn't working for me btw.. – Fadi Jul 11 '16 at 18:20
Ok i am doing a POSIX version... – Luciano Andress Martini Jul 11 '16 at 18:21
It is Better now? – Luciano Andress Martini Jul 11 '16 at 18:24
Yes.. Thank you so much for the script, this will be useful for later use! :) Once I have enough points I'll come back and +1 you.. haha – Fadi Jul 11 '16 at 18:28

sed/awk remove newline on two pattern matches

4 Answers4