37

I have a number of files I want to update by replacing one multi-line string with another multi-line string. Something along the lines of:

* Some text, 
* something else
* another thing

And I want to replace it with:

* This is completely
* different text

The result would be that after the replacement the file containing the first block of text, will now contain the second string (the rest of the file is unchanged).

Part of the problem is that I have to find the list of files to be updated in the file system. I guess I can use grep for that (though again that's not as easy to do with multiline strings) then pipe it in sed maybe?

Is there an easy way to do this? Sed is an option but it's awkward because I have to add \n etc. Is there a way to say "take the input from this file, match it in those files, then replace it with the content of this other file" ? I can use python if need be, but I want something quick and simple, so if there is a utility available, I would rather use that than write my own script (which I know how to do).

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
ventsyv
  • 1,009
  • 1
    You should probably use perl for this.

    http://stackoverflow.com/questions/1030787/multiline-search-replace-with-perl

     – orion Jan 26 '15 at 19:15
  • 3
    So you want to match some text, something else another thing whether or not it spans multiple lines? Or you only want to match some text,\nsomething else\nanotherthing? – mikeserv Jan 26 '15 at 19:16
  • 2
    Edit your question and clarify what is exactly the content of each file(s), and what the desired output is. – jimmij Jan 26 '15 at 19:25
  • The string spans multiple lines. I rather disregard the whitespace when matching/replacing because it might not all be absolutely the same, but it's not a big deal if I just do 1-1 match (newlines and all). – ventsyv Jan 26 '15 at 19:43
  • This is similar to https://unix.stackexchange.com/questions/26284/how-can-i-use-sed-to-replace-a-multi-line-string/610814#610814 – laktak Sep 22 '20 at 20:51

4 Answers4

32

Substitute "Some...\n...Thing" by the contents of file "new" in one or more input files

perl -i -p0e 's/Some.*?thing\n/`cat new`/se' input.txt ...
  1. -i to change input.txt directly
  2. -p0 slurp input file file and print it in the end
  3. s/regexp/.../s in regexp . is .|\n
  4. s/.../exp/e replace by eval(exp)
  5. new -- a file containing the replacement text (This is completely...different text)
  6. if useful you can expand the original text s/Some text\n...\n...thing\n/...
JJoao
  • 12,170
  • 1
  • 23
  • 45
  • How can I do the same with a file called say "before" to look for the (multi-line) contents of that file? I tried but it does not work. – Kvothe Dec 06 '19 at 16:56
  • @Kvothe, we need more details... Assuming that "before" has no special chars, you can try perl -i -p0e ' $b= \cat before`; s/$b/Some thing\n/se' input.txt ...` – JJoao Dec 07 '19 at 08:24
  • And assuming the "before" contains all special characters (new lines, slashes, brackets) except ' and `. – Kvothe Dec 07 '19 at 15:32
  • I get an error "Unrecognized character \xE4; marked by <-- HERE after do {<-- HERE near column 5 at -e line 2." when executing this command. There are unicode characters in the strings I am inputting. Does perl support unicode? – Lost Crotchet May 14 '20 at 00:36
  • This worked for me only if I do not use /e. I had to use only /s instead. – tekknolagi Oct 08 '20 at 03:51
  • @tekknolagi, without /e the new file would not be included. We woud get just the string cat new. – JJoao Oct 09 '20 at 07:18
6
sed -e :n -e '$!N;/\n.*\n/!{$!bn
};  s/some text,\n* *something else\n* *another thing/this is completely\
different text/;P;D' <infile

I fear you're going to have a difficult time coming up with a solution that suits you until you hammer out a concrete description of the problem - but that's what QA is best suited for, as I see it. Maybe this will give you an idea - it will always keep 3 lines in pattern space at a time - with a 2 line lookahead - while sliding forward through the input file only a line at a time.

It should be able to match your string whether it spans multiple lines or not - up to three, that is. But there are no provisions for mirroring that provision in the replacement - it always spans two lines as written.

mikeserv
  • 58,310
4

While ripgrep specifically doesn't support inline replacement, I've found that its current --replace functionality is already useful for this use case:

rg --replace "$(cat new.txt)" --passthru --no-line-number \
--multiline --multiline-dotall 'Some.*?thing\n' multi.txt > output.txt
  • --replace 'string' enables replacement mode and sets the replacement string. It can also include captured regex groups by using $1 etc.
  • $(cat new.txt) passes the contents of the file new.txt as the replacement string.
  • --passthru is needed since ripgrep usually only shows the lines matching the regex pattern. With this option it also shows all lines from the file that don't match.
  • --no-line-number / -N is because by default ripgrep includes the line numbers in the output (useful when only the matching lines are shown).
  • --multiline / -U enables multiline processing which is disabled by default.
  • --multiline-dotall is only needed if you want the dot ('.') regex pattern to match newlines (\n).
  • > output.txt is needed since inline replace isn't supported. With the --passthrough and no-line-number options the standard output matches the desired new file with replacements and can be saved as usual.

However, this command isn't as useful for processing multiple files, as it needs to be run separately per file.

Silveri
  • 161
1

Not to strong (because don't chech second string but it easy to settle) and can be is not posix compilant but very simple:

sed '/^Some text/{:1;/another thing$/!{N;b 1}
     s/.*/this is completely\ndifferent text/g}' input.txt

First command add lines from Some text until have met another thing then second line change it to other text.

NOTE Limitation is that Some text should always be followed by another thing.

Costas
  • 14,916
  • The problem is that the string might be more than 2 lines (up to a dozen or so) and may contain other stuff that might need to be escaped, such as tabs, * etc. – ventsyv Jan 26 '15 at 21:43
  • @ventsyv There is no problem with number of lines or separators - script checks the start and the end only. It is quite enought IF start string is exeptionally can mark text to change. If there is not your better show the input example to produce correct pattern. – Costas Jan 26 '15 at 22:30