How to combine multiple text files and remove duplicates effectively?

Question

I tried many things but most of them just got stuck or didn't remove the duplicates properly. It doesnt matter if the output file is sorted or not

If file1.txt contains

a
b
c
d
e
c
d

If file2.txt contains

c
d
e
a
f
g
g
h

output.txt should contain

a
b
c
d
e
f
g
h

αғsнιη · Accepted Answer · 2020-04-06T07:43:36.173

0

simply use sort -u file[12].txt >output.txt. that sorts the input while removed duplicates.

make sure your files are in Unix type newline format (LF \n) instead of Dos/Windows newline format (CRLF \r\n); if not convert them by tr -d $'\r' <filename or dos2unix filename command.

edited Apr 06 '20 at 07:43

answered Apr 06 '20 at 06:42

αғsнιη

41,407

I tried that. I used sort -u first.txt second.txt third.txt >output.txt but this is what i got. The duplicates didn't get removed – Aeden Thomas Apr 06 '20 at 07:11

How to combine multiple text files and remove duplicates effectively?

1 Answers1