11

How can I convert multi lines to a single line with spaces and quotes using awk or tr or any other tool which makes it simple (but not for loops)?

$ cat databases.txt
Wp_new
Frontend DB
DB_EXT

Expected:

$ cat databases.txt
"Wp_new" "Frontend DB" "DB_EXT"  

Edit 1:

Thanks for all the useful answers. But the one I marked as correct is the one which can be typed on terminal in short time and with fewer hassles (simplicity) so that I(syadmins) can do the operations very fast without making the systems downtime more.

Akhil
  • 1,290

11 Answers11

13

It can be done with

awk '{printf("\"%s\" ",$0)} END { printf "\n" }' databases.txt

Output:

"Wp_new" "Frontend DB" "DB_EXT" 
  • @roaima please post your solution as an answer :) – Arnab Nandy Jun 21 '20 at 11:21
  • @roaima thanks updated. – Arnab Nandy Jun 21 '20 at 12:40
  • @roaima Adding printf "\n" is wrong. It doesn't solve the trailing blank problem and it's hard-coding the value you hope ORS will have (\n) instead of simply using ORS as would occur if you used print "" instead of printf "\n". If the OP cares about an added blank or a mising newline then the right way to do this with awk is what I show at https://unix.stackexchange.com/a/594059/133219 - use the value OFS as appropriate and end with the value of ORS as appropriate. – Ed Morton Jun 21 '20 at 13:41
  • @EdMorton we're joining lines as quoted words with a trailing newline. What other value of ORS were you considering!? – Chris Davies Jun 21 '20 at 14:15
  • 1
    @roaima \r\n and \0 are common ones that come to mind but beyond that philosophically - awk has a variable ORS that contains the value of the record terminator, any time you have a variable for a specific purpose writing code that then hardcodes the value of that variable is generally bad, especially if it's lengthier code (printf "\n") than code that just uses the variable (print ""). – Ed Morton Jun 21 '20 at 14:19
9

With sed + paste

$ sed 's/.*/"&"/' databases.txt
"Wp_new"
"Frontend DB"
"DB_EXT"

$ sed 's/.*/"&"/' databases.txt | paste -sd' ' - "Wp_new" "Frontend DB" "DB_EXT"

Or, just paste (courtesy https://unix.stackexchange.com/a/593240)

$ <databases.txt paste -d '"' /dev/null - /dev/null | paste -sd' ' -
"Wp_new" "Frontend DB" "DB_EXT"


If input has empty lines that should be ignored:

$ cat ip.txt
Wp_new

Frontend DB

DB_EXT $ sed -n 's/..*/"&"/p' ip.txt | paste -sd' ' - "Wp_new" "Frontend DB" "DB_EXT"

Sundeep
  • 12,008
3
$ awk '{printf "%s\"%s\"", sep, $0; sep=OFS} END{print ""}' file
"Wp_new" "Frontend DB" "DB_EXT"
Ed Morton
  • 31,617
3

There are a number of ways of achieving the result using sed. Here are two solutions:

$ sed 's/^/"/;s/$/"/' infile | sed ':a;{N;s/\n/ /;ba}'
"Wp_new" "Frontend DB" "DB_EXT"

or

$ sed 's/.*/"&"/' infile | sed ':a;{N;s/\n/ /;ba}'
"Wp_new" "Frontend DB" "DB_EXT"

First, double quotes are added to the start and end of each text line, then the text lines are joined together.

fpmurphy
  • 4,636
  • I always prefer sed over awk. thank you. – Akhil Jun 21 '20 at 05:25
  • (1) Your sed command with the N;s/\n/ /;ba ends up with the entire input file (plus the added quotes) in the pattern space (i.e., it treats it as one very long line), right? (Or am I misreading it?) While GNU sed seems to allow the pattern space to grow as large as it can get with dynamically allocated memory, POSIX specifies only that the pattern space be able to handle 8192 bytes, so this might fail on non-GNU systems. … (Cont’d) – G-Man Says 'Reinstate Monica' Jun 21 '20 at 17:00
  • (Cont’d) …  (2) I believe that you don’t need the braces in that sed command — :a; N; s/\n/ /; ba seems to work. – G-Man Says 'Reinstate Monica' Jun 21 '20 at 17:00
  • @G-ManSays'ReinstateMonica' Yes the braces are optional in this particular case. – fpmurphy Jun 22 '20 at 01:19
2

GNU xargs

<databases.txt xargs -d'\n' printf '"%s"\n' | paste -sd\  ;

Or we can use perl also:

perl -l40 -012 -pe 's/^|$/"/g;$\=$/ if eof' databases.txt

Results:

"Wp_new" "Frontend DB" "DB_EXT"
2

It could be done in some sed with:

$ sed ':a;N;$!ba;s/\n/" "/g;s/.*/"&"/' databases.txt

"Wp_new" "Frontend DB" "" "DB_EXT" "" "empty"

Or, if you don't like branch and branch labels:

$ sed -n '/^$/!{                                          
    ${H;x;s/\n/" "/g;s/.*/"&"/;p;d;}
    $!{H;1h}
    }' databases.txt

"Wp_new" "Frontend DB" "DB_EXT" "empty"

But both load the whole file into memory, which could use a lot for long files.

With some help from paste, sed could read lines one-at-a-time:

$ sed 's/.*/"&"/' databases.txt | paste -sd ' '

"Wp_new" "Frontend DB" "" "DB_EXT" "" "empty"

There is no need for the usual '-' for paste as it reads from stdin by default when given no FILE.

It could also be done in awk with explicit values:

$ awk 'BEGIN{dq="\""; sp=""} {
     printf "%s%s%s%s", sp, dq, $0,dq; sp=" "
     }END{print ""}' databases.txt

"Wp_new" "Frontend DB" "DB_EXT"

1

Here’s another sed solution.  Inevitably, it has some overlap with the answers by fpmurphy and Sundeep.

{ sed 's/.*/"&"/; 1!s/^/ /' databases.txt | tr -d '\n'; echo;}
  • Wrap each line in quotes.
  • Add a space to the beginning of every line but the first.
  • Delete all newlines.
  • Add a newline at the end.

As Sundeep points out, you can move < databases.txt to the beginning of the command if you particularly prefer to see the input specification in that position (rather than in the middle of the command).  And you can replace the echo with printf '\n' — although, following up on Ed Morton’s comments, echo might do a better job of generating the appropriate end-of-line character(s) in something other than a pure Unix/Linux context (e.g., a Windows/GNU hybrid).

1

I love to use tr(1) to convert mult line strings into single line strings first and then work on them via sed(1) by first inserting the trailing and initial quote, then replacing all 'internal' semicolons, i.e.:

$ cat databases.txt | tr '\n' ';' | sed 's/;$/"\n/; s/^/"/; s/;/" "/g'
"Wp_new" "Frontend DB" "DB_EXT"

Of course you can choose any other character than ;, as long as it isn't contained in the the input itself.

While this should be easy to reason about, it is not necessarily portable and definitely not POSIX as correctly pointed out by @g-man-says-reinstate-monica. The issue lies in sed needing to read the whole line---here the entire file---while POSIX only requiring 8192 byte lines (1). Additionally, the input to sed must be a file (i.e. end with a newline 2).

I want to leave this trick here as it's still convenient if the above restrictions do not apply, however I want to stress it shouldn't be used as part of a script :)

ljrk
  • 571
  • 3
  • 11
  • 2
    what if file has already ; in lines ? – Akhil Jun 22 '20 at 08:05
  • 3
    @AkhilJ '[...] as long as it isn't contained in the the input itself.' More often than not I find myself in the situation that there's one character that is not in any of the lines, thus enabling me to use this simple solution. I acknowledge however, that it's not universally applicable. – ljrk Jun 22 '20 at 11:02
  • 1
    The good news is this seems to work in GNU/Linux (i.e., using GNU sed) and that it contains an arguably clever(?) trick not seen in the other answers. The bad news is that it’s not POSIX-compliant and, IMHO, not very clear. (1) The input to sed is the (entire) file, with newlines replaced by semicolons — i.e., the entire file, treated as one line, read into the pattern space all at once. (1a) While GNU sed seems to allow the pattern space to grow as large as it can get … (Cont’d) – G-Man Says 'Reinstate Monica' Jun 23 '20 at 02:26
  • (Cont’d) …  with dynamically allocated memory, POSIX specifies only that the pattern space be able to handle 8192 bytes, so this might fail on non-GNU systems for large files.  (1b) The input to sed does not contain any newlines, not even at the end.  (Obviously you understand that, as your s/;$/"\n/g command adds the required newline at the end.)  However, POSIX specifies that the input to sed must be a text file, … (Cont’d) – G-Man Says 'Reinstate Monica' Jun 23 '20 at 02:26
  • (Cont’d) …  which means it must end with a newline.  So, theoretically, this could fail on non-GNU systems for files of any size.  (2) This is about clarity, opinion and style.  (2a) For the three-line input in the question (or any three-line input, for that matter), there will be six quote characters in the output.  All the other answers insert them in the order 1 & 2, then 3 & 4, then 5 & 6.  Your answer inserts them in the order 6, then 1, then 2 & 3, then 4 & 5.   … (Cont’d) – G-Man Says 'Reinstate Monica' Jun 23 '20 at 02:26
  • (Cont’d) …  This fractured pairing is what I referred to as an arguably clever trick. Absent an explanation, it is cryptic (again, IMO). (2b) Why don’t you at least do them in the order 1, then 2 & 3, then 4 & 5, then 6? I know the reason, but you should explain it. (2c) s commands that are anchored with ^ or $ don’t need the g modifier, since they cannot be executed more than once per line. (3) UUOC. – G-Man Says 'Reinstate Monica' Jun 23 '20 at 02:28
  • @G-ManSays'ReinstateMonica' Wow, thanks for the detailed review! I've used that on Solaris 10/11 and OpenBSD as well, but they either do more than POSIX demands (as they also handle non-\n-terminated files) or I've never had to deal with very long input (not totally unlikely as well). I'll definitely update my answer addressing your comments later this day, thanks again! – ljrk Jun 23 '20 at 07:37
  • Yeah, once I start writing, I don’t know where to stop. I’m glad you appreciate my review (many people don’t). … … … … … … … … … … P.S. Have you done a search for “UUOC” yet? – G-Man Says 'Reinstate Monica' Jun 23 '20 at 21:31
  • @G-ManSays'ReinstateMonica' No problem! Oh, I honestly missed that, but tr(1) is actually one of the few UNIX programs that cannot optionally read from stdin, so you're forced to create a redirection---which in this case wouldn't really be a big benefit but (IMHO) harm readability :) – ljrk Jun 24 '20 at 06:36
1

Using Raku (formerly known as Perl_6)

raku -e 'lines>>.raku.put;' 

OR

raku -e 'lines.map(*.raku).put;' 

Sample Input:

Wp_new
Frontend DB
DB_EXT
This "quoted" ending comment

Sample Output:

"Wp_new" "Frontend DB" "DB_EXT" "This \"quoted\" ending comment"

If you play with the various commands above, you'll find that the lines command reads input lazily (stripping newlines by default), the raku (or perl) command gives you an internal 'Perlish/Rakuish' representation of the input (i.e. adds quotes), and the put command means 'print-using-terminator'.

If the above code is unmemorizable and/or too long to type, you could try 'meeting-Raku-halfway' with the following code (i.e. below returns more "Rakuish" output):

raku -e 'lines.list.raku.put;'  databases.txt  #OP's database.txt file

("Wp_new", "Frontend DB", "DB_EXT")

Or more simply:

raku -e 'lines.raku.put;'  databases.txt  #OP's database.txt file

("Wp_new", "Frontend DB", "DB_EXT").Seq

https://docs.raku.org/routine/lines
https://raku.org/

jubilatious1
  • 3,195
  • 8
  • 17
  • Note: I added the line This "quoted" ending comment to illustrate Raku's elegant handling of quote characters (escaping them). The accepted sed/paste answer returns the result "Wp_new" "Frontend DB" "DB_EXT" "This "quoted" ending comment". – jubilatious1 Sep 13 '21 at 16:23
  • 1
    great. looks simple. This is the first time I encountered raku command in text processing, and I think it's not available in POSIX like sed. But I am going to use this from now. Thanks. @jubilatious1 – Akhil Sep 14 '21 at 04:49
0

All answer are nice, I guess this depend about what output did you expect. If you want a raw line text with no additional characters or if is necessary to add some, I tried this and it works very nice to me:

Simply collapse the text:

cat yourFile | paste -sd' ' -
>item1 item2 item3

Surround item with double-quotes:

sed -n 's/..*/"&",/p' yourFile | paste -sd' ' -
>"item1", "item2", "item2",
AdminBee
  • 22,803
0
awk 'ORS=" "{print "\""$0"\""}' filename

output

"Wp_new" "Frontend DB" "DB_EXT"