Delete all files that DON'T have duplicate names?

Question

Given a large list of files, containing the following:

FILE1.doc
FILE1.pdf
FILE2.doc
FILE3.doc
FILE3.pdf
FILE4.doc

Is there a terminal command that would allow me to remove all files that do not have a duplicate name in the list? In this case...FILE2.doc and FILE4.doc?

Answer 1

Using bash, this will remove all files that don't have another file with the same name but different extension:

for f in *; do same=("${f%.*}".*); [ "${#same[@]}" -eq 1 ] && rm "$f"; done

This approach is safe for all file names, even those with white space in their names.

How it works

• for f in *; do

  This starts a loop over all files in the current directory.

• same=("${f%.*}".*)

  This creates a bash array with the names of all files with the same basename.

  $f is the name of our file. ${f%.*} is the name of the file without its extension. If, for example, the file is FILE1.doc, then ${f%.*} is FILE1. "${f%.*}".* is all the files with the same basename but any extension. ("${f%.*}".*) is a bash array of those names. same=("${f%.*}".*) assigns the array to the variable same.

• [ "${#same[@]}" -eq 1 ] && rm "$f"

  If there is only one file with this basename, we delete it.

  "${#same[@]}" is the number of files in the array same. [ "${#same[@]}" -eq 1 ] is true if there is only one such file.

  && is logical-and. It causes the statement which follows, rm "$f" to be executed only if the statement which precedes it returns logical true.

• done

  This marks the end of the for loop.

Answer 2

Suppose your list of files is in some file /tmp/files.list, e.g. after ls * > /tmp/files.list ;

Then sort -u /tmp/files.list gives you a sorted file lists without duplicate (not needed if you did the ls * > /tmp/files.list above). You could process that with some awk script inspired from this, e.g.

sort -u /tmp/files.list | awk '{
  function basename(file) {
    sub(".*/", "", file)
    return file
  }
curfil=$0;
if (basename(curfil)==basename(prevfil)) system("rm " + curfil);
prevfil=curfil;
}'

Beware, I have not tested this.

Answer 3

Another simple looking complex way can be:

for x in `for i in *; do echo $i ; done | cut -d'.' -f1 | uniq -u `; do rm $x.*; done