Remove duplicate words between lists

Question

Remove duplicate words from between lists example:

We have two lists; the first list contains:

a
b
c
d

The second list contains:

a
b
c
d
e
f

I want to make a comparison between the first and second lists, removing the matches contained in both, resulting in this:

e
f

I couldn't find a solution to do that with bash, but I did find one in python: https://stackoverflow.com/questions/7961363/removing-duplicates-in-lists/7961390#7961390

Have you tried something so far? Folks here will be more inclined to help if you show some effort. — schrodingerscatcuriosity, Oct 17 '20 at 14:16
I couldn't find a solution to do that with bash But a solution was found in Python https://stackoverflow.com/a/7961390/13079083
It's similar to what I want to do but I don't know a function like set in BASH — Notme, Oct 17 '20 at 14:25
Does this answer your question? Two files comparison in bash script? — schrodingerscatcuriosity, Oct 17 '20 at 14:31
It looks like it but the list must be similar to give correct results list a a \ nb \ nc \ nd ... list b a \ nb \ nc \ nd \ ne ... If the letters are not in order, it will give false and incomplete results Especially if you use long words in the comparison, all results will be incorrect — Notme, Oct 17 '20 at 15:24
The grep solution linked will remove all duplicate lines. The files do not need to be "similar" to each other. — markasoftware, Oct 18 '20 at 03:32

annahri · Answer 1 · 2020-10-18T09:00:35.343

You can use diff using --GTYPE-group-format=GFMT option. From man diff :

--GTYPE-group-format=GFMT
    format GTYPE input groups with GFMT
LTYPE is 'old', 'new', or 'unchanged'.
    GTYPE is LTYPE or 'changed'.
GFMT (only) may contain:


%<     lines from FILE1
%>     lines from FILE2
%=     lines common to FILE1 and FILE2

In your case, you can use diff --new-group-format='%>' --unchanged-group-format='' list1 list2

$ cat list1
a
b
c
d
$ cat list2
a
b
c
d
e
f
$ diff --new-group-format='%>' --unchanged-group-format='' list1 list2
e
f

Explanation

The --new-group-format='%>' will output any new entries from FILE2 (%>) that doesn't exist in FILE1.
The --unchanged-group-format='' will prevent diff to print any identical lines.

Remove duplicate words between lists

1 Answers1

Explanation