Group and count file names following a pattern

Question

I have a large number of files in a folder with a specific naming system. It looks somewhat like this:

my_file_A_a.txt
my_file_A_d.txt
my_file_A_f.txt
my_file_A_t.txt
my_file_B_r.txt
my_file_B_x.txt
my_file_C_f.txt
my_file_D_f.txt
my_file_D_g.txt
my_file_E_r.txt

I would like a command line, or a series of commands (can use temp files, I have write access), that would return something like:

A: 4
B: 2
C: 1
D: 2
E: 1

It could be done with a lot of ls -1 *A* | wc -l commands, but it would take a long time as there are a few hundred "groups" to count.

Also, each group name is unique. There is an A group, a B group, but no AB group.

Answer 1

Assuming that your filenames are "well-behaved", i.e. they don't contain newlines, the following combination of ls and awk would work:

ls -d my_file* | awk -F'_' 'NF==4{count[$3]++} END{for (i in count) printf "%s: %d\n", i, count[i]}'

This will redirect the output of the ls command that lists all files starting my_file* to an awk program. The awk program will use the _ as field separator and check the 3rd field to track the occurence in an array count, which uses the group number as "array index".

At the end, it prints an overview of how often each group occurred.

Notice

• There is a "minimum" safeguard against completely malformed filenames by requiring exactly 4 such fields. This assumes the _ cannot be part of the a,d,f,... part of the filenames in your example.
• The output will not necessarily be sorted according to the category names. The sorting order will depend on how awk traverses the array indices in the for (i in count) loop. If sorting is desired, you can add a further pipe to sort. Alternatively, if you use GNU Awk, you can add a configuration setting via

  BEGIN{PROCINFO["sorted_in"]="@ind_str_asc"}
  

  before the NF==4{...} rule. This will ensure arrays are traversed according to the array index, sorted in lexicographical (ASCII) order.
• This will work with the limitation stated at the beginning, and because your file name structure is rather simple. In general it is discouraged to parse the output of ls.

Answer 2

for f in my_file_*_*.txt
do
    f="${f#my_file_}"
    printf "%s\n" "${f%%_*.txt}"
done |
sort |
uniq -c

The for loop reformats each filename f to strip off the leading my_file_ and the trailing _whatever.txt, then sorts that output, and uses uniq to count the number of occurrences of each unique value.

Answer 3

I would approach it with a loop over a wildcard, then extract the field from the filename with bash's regular expression feature in its [[ Conditional Expression construct.

unset collect
declare -A collect
for f in ./*_*_*_*.txt
do 
  [[ $f =~ [^_]+_+[^_]+_+([^_]+)_+[^_]+.txt ]] &&
  ((collect["${BASH_REMATCH[1]}"]++))
done
for group in "${!collect[@]}"
do
  printf '%s: %d\n' "$group" "${collect["$group"]}"
done

The only parenthesized field is the 3rd underscore-delimited one; once it's captured, we increment that value in an associative array (collect).

Answer 4

Using Raku (formerly known as Perl_6)

raku -e '.say for dir.split("_")[2,5,8...*].Bag.pairs.sort;'

Sample Input (current directory listing):

my_file_A_a.txt
my_file_A_d.txt
my_file_A_f.txt
my_file_A_t.txt
my_file_B_r.txt
my_file_B_x.txt
my_file_C_f.txt
my_file_D_f.txt
my_file_D_g.txt
my_file_E_r.txt

Sample Output:

A => 4
B => 2
C => 1
D => 2
E => 1

As a brief explanation, the current directory dir() listing is obtained and split on _ underscore. [File names are assumed not to start/end with _ underscore]. The elements obtained are thus:

raku -e 'dir.split("_").raku.say;'
("my", "file", "A", "a.txt my", "file", "A", "d.txt my", "file", "A", "f.txt my", "file", "A", "t.txt my", "file", "B", "r.txt my", "file", "B", "x.txt my", "file", "C", "f.txt my", "file", "D", "f.txt my", "file", "D", "g.txt my", "file", "E", "r.txt").Seq

After that, Raku has a fairly robust mechanism for generating/understanding sequences: simply typing in [2,5,8...*] lets you pull out the letters A,B,C,D,E (every third element, numbering starts from 0). Then Bag, pairs, and sort.

(If you're sure you have no blank spaces in your file names, you could add a second call to split(" ") after the first one. Then the elements you would pull out would be [2,6,10...*] ).

NOTE 1: If you have extraneous file names that don't fit the pattern listed by the OP (and are mucking up your counts), then you could change the dir call to something like dir(test => / [ <-[_]>+ _ ] ** 3 /) which subsets filenames on a regex where one-or-more non-underscores are followed by an underscore, repeated three times.

NOTE 2: If you want two columns of output (no => in-between), simply change .say to .put. Or if you prefer a more 'Raku-ish' output, try using .raku.say, which returns the following:

:A(4)
:B(2)
:C(1)
:D(2)
:E(1)

https://docs.raku.org/routine/dir
https://docs.raku.org/type/Bag
https://raku.org

Answer 5

A filename containing four underscore-delimited fields and ending with the string .txt is matched by the extended globbing pattern +([!_])_+([!_])_+([!_])_+([!_]).txt. Each +([!_]) matches one or more non-underscore characters, just like [^_]+ would do as an extended regular expression.

We can extract the third field from this by removing the initial two fields and the last field along with the .txt suffix string.

#!/bin/bash
shopt -s extglob nullglob
names=( +([!_])+([!])+([!])+([!]).txt )
names=( "${names[@]#+([!_])+([!])}" )
names=( "${names[@]%+([!_]).txt}" )
printf '%s\n' "${names[@]}" | sort | uniq -c

The script only assumes that the third field in the filename does not contain embedded newlines.

Testing this on the example filenames in the question:

$ ls
list              my_file_A_f.txt   my_file_B_x.txt   my_file_D_g.txt
my_file_A_a.txt   my_file_A_t.txt   my_file_C_f.txt   my_file_E_r.txt
my_file_A_d.txt   my_file_B_r.txt   my_file_D_f.txt   script
$ ./script
   4 A
   2 B
   1 C
   2 D
   1 E

You could filter this through a simple awk script to get it into whatever format you wish.

$ ./script | awk '{ printf "%s: %d\n", $2, $1 }'
A: 4
B: 2
C: 1
D: 2
E: 1

---

If your names are well-behaved, meaning there are no embedded newline characters at all in any of them, then you may simplify the script somewhat and use cut instead.

#!/bin/bash
shopt -s extglob nullglob
printf '%s\n' +([!_])+([!])+([!])+([!]).txt |
cut -d _ -f 3 | sort | uniq -c

Answer 6

Sort, sed and uniq enough:

ls |grep my_file | sed "s/.*_.*_\(.*\)_.*txt/\1/"|sort |uniq -c|sed "s/[^0-9]*\([0-9]*\) \(.*\)/\2: \1/"

Another oneliner, just 3 variables:

count=0;chchange="dummy";ls | sed -n "s/.*my_file.*_\(.*\)_.*txt/\1/p"|sort|cat - <(echo end) |while read a ; do  if [ $a == $chchange ] ; then  ((count++));else if [ $chchange != "dummy" ] ;then  echo "$chchange $count"; fi; count=1; chchange=$a; fi;  done;

Need to put one extra line to the sort output.