How can I compare different files without opening them?

Question

I have two directories A and B; each one contains a lot of sub-directories

geom001, geom002 ....etc

each sub-directory contains a file named results. I want to compare, without opening any of them, each file in A with each file in B and find if there is a file or more in A similar to one or more file in B. How can I use command like the following in a loop to search over all files?

cmp --silent  file1 file2  || echo "file1 and file2 are different"

Answer 1

If files are exactly the same, then their md5sums will be exactly the same, so you can use:

find A/ B/ -type f -exec md5sum {} + | sort | uniq -w32 -D

An md5sum is always exactly 128 bits (or 16 bytes or 32 hex digits) long, and the md5sum program output uses hex digits. So we use the -w32 option on the uniq command to compare only the first 32 characters on each line.

This will print all files with a non-unique md5sum. i.e. duplicates.

NOTE: this will detect duplicate files no matter where they are in A/ or B/ - so if /A/subdir1/file and A/subdir2/otherfile are the same, they will still be printed. If there are multiple duplicates, they will all be printed.

You can remove the md5sums from the output by piping into, e.g., awk '{print $2}' or with cut or sed etc. I've left them in the output because they're a useful key for an associative array (aka a 'hash') in awk or perl etc for further processing.

Answer 2

The seeming challenges of the question / request is perhaps the recursion aspect.

Assuming that cmp is an adequate utility and that both folder / directories 1 & 2 to be compared are of the same structure (ie same files & folders) and reside within the same root path - you can try something similar to:

#!/bin/bash
ROOT=$PWD ; # #// change to absolute path eg: /home/aphorise/my_files
PWD1="1/*" ;
PWD2="2/*" ;

# #// Get lengths of seperators
IFS=/ read -a DP <<< ${ROOT} ;
PLEN1=${#DP[*]} ;
IFS=/ read -a DP <<< ${PWD1} ;
PLEN1=$(echo "${#DP[*]}" + $PLEN1 - 1 | bc) ;
IFS=/ read -a DP <<< ${PWD2} ;
PLEN2=${#DP[*]} ;

# #// Set absolute paths:
PWD1="${ROOT}/${PWD1}" ;
PWD2="${ROOT}/${PWD2}" ;
DIFFS=0 ;

function RECURSE()
{
    for A_FILE in $1 ; do
        if [ -d $A_FILE ] ; then
            RECURSE "$A_FILE/*" ;
        else
            IFS=/ read -a FP <<< ${A_FILE} ;
            B_FILE="${PWD2:0:${#PWD2}-${#PLEN2}}$( IFS=/ ; printf "%s" "${FP[*]:$PLEN1:512}"; )" ;
            if ! cmp ${A_FILE} ${B_FILE} 1>/dev/null ; then printf "$A_FILE --> $B_FILE <-- DIFFER.\n" ; ((++DIFFS)) ; fi ;
        fi ;
    done ;
}

printf "Starting comparison on $PWD1 @ $(date)\n\n" ;
RECURSE "${PWD1[*]}" ;
if ((DIFFS != 0)) ; then printf "\n= $DIFFS <= differences detected.\n" ; fi ;
printf "\nCompleted comparison @ $(date)\n" ;

UPDATE:

Following with another script - subsequent to additional feedback received - to unconditionally compare all files in directory 1 with 2:

#!/bin/bash
PWD1="$PWD/1/*" ;
PWD2="$PWD/2/*" ;
DIFFS=0 ;
NODIFFS=0 ;

printf "Starting comparison on $PWD1 @ $(date)\n\n" ;

FILES_A=$(find ${PWD1} -type f) ;
FILES_B=$(find ${PWD2} -type f) ;

for A_FILE in ${FILES_A[*]} ; do
        for B_FILE in ${FILES_B[*]} ; do
                if ! cmp ${A_FILE} ${B_FILE} 1>/dev/null ; then
                        printf "$A_FILE & $B_FILE <- DIFFER.\n" ;
                        ((++DIFFS)) ;
                else
                        printf "\n-> SAME: $A_FILE & $B_FILE\n" ;
                        ((++NODIFFS)) ;
                fi ;
        done ;
done ;

printf "\n= $DIFFS <= differences detected - & - $NODIFFS <= exact matches.\n" ;
printf "\nCompleted comparison @ $(date)\n" ;

Answer 3

I think this will get you close. It will list out the cmp output for all files named results in A compared to all files named results in B.

find ./A -name results | xargs -I REPLACESTR find ./B -name results -exec cmp REPLACESTR {} \;