while read line find – faster with perl?

Question

I have a text file with three columns separated by tab and I read the third column line by line to find all files in a directory which have this in the name. Since it is a file with up to 1000 entries, my attempt to solve it with "find" is not suitable because it takes too much time.

while read f; 
do var1=`echo "$f" | cut -f1`; 
var2=`echo "$f" | cut -f2` ; 
var3=`echo "$f" | cut -f3`; 
echo "\n ID1 = $var1 \n ID2 = $var2 \n\n Path:";
find //myDirectory/ -type f -name *$var3* -not -path '*/zz_masters/*' -exec ls -Sd {} + ;
echo "\n----------------------"; 
done >> /SearchList.txt < /ResultList.txt

As you can see, one folder is excluded and the results are sorted by size because some files are in different resolutions.

Searchlist.txt:

a1 a    1 x1    Trappist
b2 b    2 y2    Mars
c3 c    3 z3    Pegasi

Result:

/myDirectory/
ID1 = a1 a 
 ID2 = 1 x1
Path:
/myDirectory/xx/Trappist-1.png
/myDirectory/xx/Trappist-2.png

ID1 = b2 b 
 ID2 = 2 y2
Path:
/myDirectory/yy/Mars-1.jpg

ID1 = c3 c 
 ID2 = 3 z3
Path:
/myDirectory/xx/51PegasiB.tif

In the hope that it works faster, I tried it with perl. I am new to perl but my results are sad and I am stuck in the script. It creates a loop . That's where I'm at:

perl find.pl /myDirectory/ /SearchList.txt /ResultList.txt
#!/usr/bin/perl -w
use strict; 
use warnings; 
use File::Find;
open (IN, "$ARGV[1]") or die;
open(my $fh_out, '>', "$ARGV[2]");
my @files;
print $fh_out "$ARGV[0]\n";
while (my $line = <IN>) {

    chomp $line;
my @columns = split(/\t/, $line);
find(sub { 
      push @files,"$File::Find::name" if /$columns[2]/;
I think print has to be inside sub but each search result  shows separately and is still slow:
print $fh_out "\n\n----------------------------\n
#ID1: $columns[0]\nID2: $columns[1]Searchstring: $columns[2]\n
#Path:\n", "$File::Find::name\n" if /$columns[2]/;
}, $ARGV[0]);


outside sub: displays the search results together, but also slow and with a loop :(
print $fh_out "\n\n----------------------------\n
ID1: $columns[0]\nID2: $columns[1]
Searchstring: $columns[2]\n\nPath:\n", join "\n", @files;
}
close IN;
close $fh_out;
exit;

Will perl possibly not give the speed increase I want, and if not, what alternatives would there be?

Answer 1

A code review of your bash code:

• read can pick out the words for you
• echo "\n" won't print a newline
• use $(...) instead of `...` - ref
• use proper indentation Be more careful with your redirection symbols

while read -r var1 var2 var3 rest; do
    printf "\n ID1 = %s \n ID2 = %s \n\n Path:\n" "$var1" "$var2"
    find //myDirectory/ -type f -name "*$var3*" -not -path '*/zz_masters/*' -exec ls -Sd {} +
    # ........................ quoted ^.......^
    printf "\n----------------------\n"; 
done < /SearchList.txt > /ResultList.txt

However the way to speed this up is to only run find once:

id1=()
id2=()
substrings=()
names=( -false )
declare -A paths=()
while read -r var1 var2 var3 rest; do
    id1+=( "$var1" )
    id2+=( "$var2" )
    substrings+=( "$var3" )
    names+=( -o -name "$var3" )
done < /SearchList.txt
find /myDirectory/ -type f ( "${names[@]}" ) -not -path '/zz_masters/' -prinf "%s %p\0" 

| sort -znr 

| while read -d '' -r size name; do
    for s in "${substrings[@]}"; do
        if [[ $name == "$s" ]]; then
            paths[$s]+="$name"$'\n'
            break
        fi
    done
done
fmt="\n ID1 = %s \n ID2 = %s \n\n Path:\n%s\n----------------------\n"
for idx in "${!id1[@]}"; do
    printf "$fmt" "${id1[idx]}" "${id2[idx]}" "${paths[${substrings[idx]}]}"
done > /ResultList.txt

Answer 2

You could try this if your file names don't contain tabs or newlines:

find . -type f -print |
awk '
    NR==FNR {
        name2ids[$3][1] = $1
        name2ids[$3][2] = $2
        next
    }
    {
        for (name in name2ids) {
            if ( index($NF,name) ) {
                matches[name][$0]
            }
        }
    }
    END {
        for (name in name2ids) {
            print "ID1 =", name2ids[name][1]
            print "ID2 =", name2ids[name][2]
            print "\nPath:"
            if (name in matches) {
                for (file in matches[name]) {
                    print file
                }
            }
        }
    }
' FS='\t' SearchList.txt FS='/' -

The above uses GNU awk for arrays of arrays, here is a POSIX version (untested):

find . -type f -print |
awk '
    NR==FNR {
        name2ids[$3] = $1 RS $2
        next
    }
    {
        for (name in name2ids) {
            if ( index($NF,name) ) {
                matches[name] = (name in matches ? matches[name] RS : "") $0
            }
        }
    }
    END {
        for (name in name2ids) {
            split(name2ids[name],ids,RS)
            print "ID1 =", ids[1]
            print "ID2 =", ids[2]
            print "\nPath:"
            split(matches[name],files,RS)
            for (idx in files) {
                print files[idx]
            }
        }
    }
' FS='\t' SearchList.txt FS='/' -