Remove lines based on duplicates within one column without sort

Question

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a bit cumbersome, and I can't use something like the below code because the entire lines aren't identical; just the contents of column 3.

awk '!seen[$0]++' filename

Answer 1

Just change your awk command to the column you want to remove duplicated lines based on that column (in your case third column):

awk '!seen[$3]++' filename

This command is telling awk which lines to print. The variable $3 holds the entire contents of column 3 and square brackets are array access. So, for each third column of line in filename, the node of the array named seen is incremented and the line printed if the content of that node(column3) was not (!) previously set. By doing this, always the first lines (unique by the third column) will be kept.

Above will work if your columns in input file are delimited with Spaces/Tabs, if that is something else, you will need to tell it to awk with its -F option. So, for example if columns delimited with comma(,) and wants to remove lines base on third column, use the command as following:

awk -F',' '!seen[$3]++' filename

Answer 2

sort command is already optimized to handle huge files. So, you could very well use the sort command on your file as,

sort -u -t' ' -k3,3 file

• -u - print only the unique lines.
• -t - specify the delimiter. Here in this example, I just use the space as delimiter.
• -k3,3 - sort on 3rd field.

You could refer to this answer which suggests that GNU sort is in fact the better approach for sorting big files. In your case, I think even without -parallel, you could achieve your end result without much of time delay.