Is there a tool to get the lines in one file that are not in another?

Question

Is there any tool that can get lines which file A contains, but file B doesn't? I could make a little simple script with, e.g, perl, but if something like that already exists, I'll save my time from now on.

Answer 1

Yes. The standard grep tool for searching files for text strings can be used to subtract all the lines in one file from another.

grep -F -x -v -f fileB fileA

This works by using each line in fileB as a pattern (-f fileB) and treating it as a plain string to match (not a regular regex) (-F). You force the match to happen on the whole line (-x) and print out only the lines that don't match (-v). Therefore you are printing out the lines in fileA that don't contain the same data as any line in fileB.

The downside of this solution is that it doesn't take line order into account and if your input has duplicate lines in different places you might not get what you expect. The solution to that is to use a real comparison tool such as diff. You could do this by creating a diff file with the context value at 100% of the lines in the file, then parsing it for just the lines that would be removed if converting file A to file B. _{(Note this command also removes the diff formatting after it gets the right lines.)}

diff -U $(wc -l < fileA) fileA fileB | sed -n 's/^-//p' > fileC

Answer 2

The answer depends a great deal on the type and format of the files you are comparing.

If the files you are comparing are sorted text files, then the GNU tool written by Richard Stallman and Davide McKenzie called comm may perform the filtering you are after. It is part of the coreutils.

Example

Say you have the following 2 files:

$ cat a
1
2
3
4
5

$ cat b
1
2
3
4
5
6

Lines in file b that are not in file a:

$ comm <(sort a) <(sort b) -3
    6

Answer 3

from stackoverflow...

comm -23 file1 file2

-23 suppresses the lines in file2 (-2) and the lines that appear in both (-3), leaving only the unique lines from file1. The files have to be sorted (they are in your example) but if not, pipe them through sort first.

See the man page here

-1 suppress column 1 (lines unique to FILE1)
-2 suppress column 2 (lines unique to FILE2)
-3 suppress column 3 (lines that appear in both files)

Answer 4

The grep and comm (with sort) methods take a long time on large files. SiegeX and ghostdog74 shared two great awk methods for extracting lines unique to one of two files over on Stack Overflow:

$ awk 'FNR==NR{a[$0]++}FNR!=NR && !a[$0]{print}' file1 file2

$ awk 'FNR==NR{a[$0]++;next}(!($0 in a))' file1 file2

Answer 5

If the files are big and you don't have a custom order to your entries, grep takes much too long. A quick alternative would be

sort file1 > 1 
sort file2 > 2 
diff 1 2 | grep "\>" | sed -e 's/> //'

[file2-file1 results to screen, pipe to file etc.]

Changing > to < would get the opposite subtraction. rm 1 2

Answer 6

combine from moreutils package is very intuitive:

combine fileA not fileB

The files do not need to be sorted and the order is preserved.

There are also xor, or and and operators.

Answer 7

You could also consider vimdiff, it highlights the differences between files in a vim editor

Answer 8

    awk 'NR==FNR{a[$1];next}!($1 in a){print }' fileA fileB



     cat fileA
    1
    2
    3
    4
    5
    6
    7
    8
    9

  file B
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19

    output

    10
    11
    12
    13
    14
    15
    16
    17
    18
    19


    It will print contents of fileB which is not in fileA