remove duplicate files

Question

On my Mac OS X 10.6.7, iTunes duplicated every single file in my music library. Now I have 3,840 files instead of 1,920. The problem is that iTunes didn't simply duplicate the whole folder, but each file in each folder, adding a 1 to the duplicate. It looks like this:

├── album1
│   ├── track1.mp3
│   ├── track1 1.mp3
│   ├── track2.mp3
│   └── track2 1.mp3
└── album2
    ├── track1.m4a
    └── track1 1.m4a

Now rather than clicking on 1,920 duplicates, I want to enter some smart line in the Terminal to get rid of the extra files:

find path/to/music/ -name "* 1.*" -delete

Though this will work in most cases, it will mess up things in the following situation:

└── album2
    ├── track 1.mp3
    └── track 1 1.mp3

Maybe I should exclude "* 1 1.*" and later on rename those files removing the extra 1?

How do I do this?

Is there a simple solution to this?

You are right — see edit! I am asking this on UNIX & LINUX because I think it's more likely you guys know... — AvL, Apr 08 '13 at 14:14
Nothing wrong with asking here about using the OS X terminal, as OS X is a Unix system. It just helps if you specify the environment because otherwise people are likely to assume that you are on Linux, with a GNU userland. (The mention of iTunes jarred with this, which prompted my request for clarification.) — user, Apr 08 '13 at 14:23

score 5 · Answer 1 · answered Apr 11 '13 at 06:46

I'd also add that to successfully find (and remove) exact file duplicates, you have to compare files by hash:

#!/bin/sh -eu
find "${1:-.}" -type f ! -empty -print0 | xargs -0 md5 -r | \
    awk '$1 in a{sub("^.{33}","");printf "%s\0",$0}a[$1]+=1{}' | \
    xargs -0 rm -v --

Save this as deldupes.sh and run it giving dirname as 1st parameter (otherwise $PWD would be used).

Tested on OS X, works for long whitespaced filenames.

score 4 · Accepted Answer · edited Apr 10 '13 at 22:28

Something like this, haven't tested; it's in bash so you may have to convert some syntax:

IFS=$'\n'              # so that only newlines separate words, not spaces
set -f                 # disable globbing
FILES=$(find path/to/music/ -name "* 1.*")

for FILE in ${FILES}; do
    if [[ -f "${FILE% 1.*}" ]] ; do
        echo "Matched ${FILE}."
        # rm "${FILE}" # Uncomment me once you have confirmed it would do what you intend.
    fi
done

${FILE% 1.*} strips the last match of the 1.* syntax from the end, [[ -f ... ]] checks whether that file would exist; therefore, it would remove the files for which the file without the syntax at the end would exist. Please test before uncommenting rm, to be sure it is correct.

Thanks a lot! Actually I simply tested if there would be a conflict with find . -name "* 1 1.*" -print. This listed 5 files, which I manually renamed after cleaning the folder with: find . -name "* 1.*" \! $ -name "* 1 1.*" $ -delete — AvL, Apr 08 '13 at 14:49

remove duplicate files

2 Answers2

Linked