Find in a given directory, if there exist 2 files that have exactly the same content irrespective of their names

Question

if the problem could be solved using a series of commands on the command line, it would be better for me than writing a script

Hi and welcome to Unix Stack Exchange. I think to do this, you would need to use the diff command. You would probably have to construct a for loop to compare each file in the directory to every other. Could do it with a fairly simple script. — Time4Tea, Jan 28 '19 at 18:05

score 1 · Answer 1 · answered Jan 28 '19 at 18:11

for x in *; do for y in *; do [ "$x" = "$y" ] && continue; cmp -s "$x" "$y" && echo Same: "$x" and "$y"; done; done|head -1

Or, broken up a bit for readability:

for x in *
do 
  for y in *
  do 
    [ "$x" = "$y" ] && continue
    cmp -s "$x" "$y" && echo Same: "$x" and "$y"
  done
done | head -1

The head is just to keep the mirror reporting down ("a = b" and "b = a").

DopeGhoti · Answer 2 · 2019-01-28T20:17:29.260

1

find . -type f -exec md5sum "{}" \; | awk 'seen[$1] { print "Duplicate file "$2" with hash "$1" at "seen[$1]" } ! seen[$1] {seen[$1]=$2}'

edited Jan 28 '19 at 20:17

answered Jan 28 '19 at 18:22

DopeGhoti

It says "Duplicate file xxx with hash yyy". That's right. But duplicate to what file? But nicely scripted! – Freddy Jan 28 '19 at 18:55
Tweaked to store the first filename found in the array for reference in the output. – DopeGhoti Jan 28 '19 at 20:17

2 Answers2