How can I fix lines broken in wrong places?

Question

My text file looks like this:

This is one
sentence that is broken.
However this is a good one.
And this
one is
somehow, broken into
many.

I want to remove the trailing newline character for any line which is followed by a line starting with a lowercase letter.

So this should be:

This is one sentence that is broken.
However this is a good one.
And this one is somehow, broken into many.

How can I do this?

Edit: There are some really good answers here, but I chose to accept the first one that worked and was earliest. Thanks so much everyone!

Answer 1

With awk:

awk -v ORS= '{print (NR == 1 ? "" : /^[[:lower:]]/ ? " " : RS) $0}
             END {if (NR) print RS}'

That is, do not append the record separator to each line (ORS empty). But prepend a record separator before the current line if not on the first line and the current line doesn't start with a lowercase letter. Otherwise prepend a space character instead, except on the first line.

Answer 2

try

awk '$NF !~ /\.$/ { printf "%s ",$0 ; next ; } {print;}' file

where

• $NF !~ /\.$/ match line where last element do not end with a dot,
• { printf "%s ",$0 print this line with a trailling space, and no line feed,
• next ; } fetch next line,
• {print;} and print it.

I am sure there will be a sed option.

Note: this will work with line ending in a dot, however condition in sentences beginning with upper case letter won't get merged. See Stéphane Chazelas's answer.

Answer 3

In perl:

#!/usr/bin/perl -w
use strict;
my $input = join("", <>);
$input =~ s/\n([a-z])/ $1/g;
print $input;

Technically you wanted to replace "newline followed by lower-case letter" with "space and-that-lower-case-letter", which is what the core of the above perl script does:

1. Read in the input to a string input.
2. Update the input variable to be the result of the search & replace operation.
3. Print the new value.

Answer 4

With sed you could use a N;P;D cycle (so as to always have two lines in the pattern space and if the first character after the newline is lowercase then replace the newline with a space) and a test - that way after each substitution you restart the cycle:

sed -e :t -e '$!N;/\n[[:lower:]]/s/\n/ /;tt' -e 'P;D' infile

Answer 5

Another way you can do this is:

perl -lpe '$\ = /\.$/ ? $/ : $"' data

wherein: $\ => ORS, $/ => IRS= \n, $" = space

perl -pe '$_ .= <>, eof or redo if s/[^.]\K\n/ /' data

sed -e '
   :a
      /\.$/!N
      s/\n/ /
   ta
' data

Answer 6

Using sed and fmt:

$ sed -e '1n; s/^[[:upper:]]/\n&/' input.txt | fmt
This is one sentence that is broken.

However this is a good one.

And this one is somehow, broken into many.

The sed script inserts a newline before every line that begins with a capital letter (except for the very first line of input). sed's output is then piped into fmt to reformat the resulting paragraphs.

Alternatively use par if you have it installed. It's another paragraph reformatter, but much more capable than fmt, with many more features and options.

Note that there will be a blank line between each paragraph. Paragraphs should be separated from each other by at least one blank line. Without the blank lines, your entire input sample is reformatted as a single multi-sentence paragraph, e.g.:

$ fmt input.txt
This is one sentence that is broken.  However this is a good one.
And this one is somehow, broken into many.

If you need to remove the blank lines after reformatting just pipe it through sed again - but this will remove ALL blank lines, including any that may have been in the original input. e.g.

$ sed -e '1n; s/^[[:upper:]]/\n&/' input.txt | fmt | sed -e '/^$/d'
This is one sentence that is broken.
However this is a good one.
And this one is somehow, broken into many.

Answer 7

Python 3

import re
print(re.sub(r'\n([a-z])', r' \1', open('file.txt').read(), flags=re.MULTILINE))

This is the same regex/substitution as Jeff's answer