Remove duplicate lines with a twist

Question

Okay so I want remove duplicate lines but it's a bit more complicated than that..

I have a file named users.txt, example of file is:

 users:email@email.com
 users1:email@email.com

Now due to a bug in my system people were able to register with the same email as someone else, so I want to remove if lines have the same email more than once, example of issue:

 user:display:email@email.com
 user2:email@email.com
 user3:email@email.com
 user4:email@email.com

Notice how user, user2, user3, user4 all have the same email.. well I want to remove user2, user3, user4 but keep user.. or vice versa ( first one to be picked up by request ) remove any other lines containing same email..

so if

 email@email.com is in 20 lines remove 19
 spam@spam.com is in 555 lines remove 554
 and so fourth..

Answer 1

This calls for Awk. Since the field you want to check is the first field of each line, just reference $1.

awk -F: '! ($1 in seen) {print; seen[$1]}' users.txt

---

You can "golf" this to reduce it considerably:

awk -F: '!a[$1]++' users.txt

The longer form is more or less self-explanatory; you build an associative array using each email address as an index, without bothering to assign a value. Then you can just check if the email address has been "seen" before (i.e., if the associative array has a particular email address as an index already), and print the whole line if not.

The shorter form is actually doing more or less the same thing, but requires more explanation for the shorter code.

The postfix ++ operator acts on a variable after the expression is evaluated, so we'll come back to that later.

In Awk, 0 means false and non-zero means true. ! is for negation and reverses the truth value.

Appearing as it does outside of curly brackets, the expression is interpreted as a boolean expression, with an associated action (in curly brackets) to be performed if the expression is true. Since no action is explicitly stated, the default (implicit) action of printing the whole line is used, if the expression evaluates to true (non-zero).

Essentially, this retrieves the value in the associative array a which is pointed to be the email address (first field) as its index—or creates that value initialized as 0 if not already present, interprets a 0 as false or non-zero as true, inverts this truth value and prints the whole line if the result is "truthy," and then increments the value stored in the associative array at that point.

A common enough Awk idiom, actually, but I wouldn't fault you for using the longer more explicit version. :)

Answer 2

1. Use GNU datamash to group input by the 2nd field, and keep only the first line of each grouping:

   datamash -t':' -g 2 rmdup 2 < users.txt
   

2. As a comment from don_crissti notes, sort can do it, but while it returns the desired results, it may also reorder the output:

   sort -t':' -k 2,2 -u users.txt
   

The above code assumes users.txt is sorted by the 2nd field, then the first field.