find common lines between multiple files

Question

I have 4 files which are like

       file A
       >TCONS_00000867
       >TCONS_00001442
       >TCONS_00001447
       >TCONS_00001528
       >TCONS_00001529
       >TCONS_00001668
       >TCONS_00001921
   file b
   >TCONS_00001528
   >TCONS_00001529
   >TCONS_00001668
   >TCONS_00001921
   >TCONS_00001922
   >TCONS_00001924

   file c
   >TCONS_00001529
   >TCONS_00001668
   >TCONS_00001921
   >TCONS_00001922
   >TCONS_00001924
   >TCONS_00001956
   >TCONS_00002048

   file d
   >TCONS_00001922
   >TCONS_00001924
   >TCONS_00001956
   >TCONS_00002048

All files contain more than 2000 lines and are sorted by first column.

I want to find common lines in all files. I tried awk and grep and comm but not working.

Answer 1

Since the files are already sorted:

comm -12 a b |
  comm -12 - c |
  comm -12 - d

comm finds common lines between files. By default comm prints 3 TAB-separated columns:

1. The lines unique to the first file,
2. The lines unique to the second file,
3. The lines common to both files.

With the -1, -2, -3 options, we suppress the corresponding column. So comm -12 a b reports the lines common to a and b. - can be used in place of a file name to mean stdin.

Answer 2

cat a b c d |sort |uniq -c |sed -n -e 's/^ *4 \(.*\)/\1/p'

Answer 3

comm would be cool if it took 3+ files as input. Since it doesn't I don't see the value in it, and would recommend just using grep:

• Doesn't require pre-sorted input
• Don't have to suppress output columns
• Command is just as short and easy as comm

grep -f a.txt b.txt | 
  grep -f - c.txt |
  grep -f - d.txt

-f <file> tells grep to look in <file> for patterns, so any line in a.txt also found in b.txt will be output as a match. This continues down the pipeline, so lines found in both a.txt AND b.txt will be used as patterns to match in c.txt. At the end of the pipeline the only output shown are lines which appear in every file.

Answer 4

Using Raku (formerly known as Perl_6)

~$ raku -e 'my %h;  for dir(test => / file \w /) {   \
               %h{$_}++ for .lines.unique };         \
            .put if .value == 4 for %h;'
#OR
~$ raku -e 'my %h; my @a = dir(test => / file \w /);  

            for @a { %h{$_}++ for .lines.unique };    

            for %h { .put if .value == @a.elems };'

Above are answers written in Raku, a member of the Perl-family of programming languages. Briefly, Raku's dir function is used to inspect the local directory and pull out regex matches to filenames. Here we assume the files are named file followed by \w a word character, but it's equally easy to (let's say) match \.txt files with the appropriate regex (not globbing pattern).

For both answers above a %h hash is declared. Once filenames are obtained (actually .IO objects), they are iterated through linewise using for, incrementing the hash key every time an identical line is seen.

In the first answer (last statement) the lines are returned if their value matches 4, i.e. from all four input files. In the second answer (last statement) the lines are returned if their value matches @a.elems, i.e. the number of input files (in other words--this value is set dynamically).

Sample Input (Note: files are named fileA, fileB, etc. Also, fileA has an extra line at the end, compared to OP's first file):

       fileA
       >TCONS_00000867
       >TCONS_00001442
       >TCONS_00001447
       >TCONS_00001528
       >TCONS_00001529
       >TCONS_00001668
       >TCONS_00001921
       >TCONS_00001922
   fileB
   >TCONS_00001528
   >TCONS_00001529
   >TCONS_00001668
   >TCONS_00001921
   >TCONS_00001922
   >TCONS_00001924

   fileC
   >TCONS_00001529
   >TCONS_00001668
   >TCONS_00001921
   >TCONS_00001922
   >TCONS_00001924
   >TCONS_00001956
   >TCONS_00002048

   fileD
   >TCONS_00001922
   >TCONS_00001924
   >TCONS_00001956
   >TCONS_00002048

Sample Output (showing all key/value counts):

~$ raku -e 'my %h; for dir(test => / file \w /) { %h{$_}++ for .lines.unique }; .say for %h.sort: -*.value;'
>TCONS_00001922 => 4
>TCONS_00001668 => 3
>TCONS_00001921 => 3
>TCONS_00001924 => 3
>TCONS_00001529 => 3
>TCONS_00002048 => 2
>TCONS_00001528 => 2
>TCONS_00001956 => 2
>TCONS_00000867 => 1
>TCONS_00001442 => 1
>TCONS_00001447 => 1

Sample Output (showing only lines where .value == 4 or .value == @a.elems):

~$ raku -e 'my %h; my @a = dir(test => / file \w /); for @a { %h{$_}++ for .lines.unique }; for %h { .key.put if .value == @a.elems};'
>TCONS_00001922

---

Finally, for those who prefer shell-globbing, Raku can do this as well. The key is remembering that the $*ARGFILES dynamic variable must be converted to .handles to read the input properly:

~$ raku -e 'my ($n,%h); for $*ARGFILES.handles -> $fh { $n++; %h{$_}++ for $fh.lines.unique }; for %h { .key.put if .value == $n };' file?
>TCONS_00001922

---

NOTE: The OP's test input seems like a trick question because no lines are in common between all four files! Thus the first file (fileA) has been modified to provide a positive control: >TCONS_00001922.

https://stackoverflow.com/a/68774047/7270649
https://docs.raku.org/routine/dir
https://docs.raku.org
https://raku.org