How do I loop over the first several files in directory?

Question

Currently I'm using the following zsh-snippet to select small batches of files for further processing

for f in $(ls /some/path/*.txt | head -2) ; do
  echo unpacking $f
  ./prepare.sh $f && rm -v $f
done

Is there a better alternative to $(ls ... | head -2) in zsh?

General overview of my task. I'm creating a data set to train a neural network. Details of that ML task are not important here. The task of the dataset creation requires me to manually process a large bunch of files. To do it, I've copied them to a separate directory. Then I randomly choose several files (first two from ls output in this example), call some preprocessing routine, review its results, move some of them to the data set being created and remove the rest. After this cleanup I again execute the command above.

Additionally, I'd like to improve my skills in shell programming and learn something new :)

The order in which these "first" files are chosen does not matter, since all of them will be processed in the end.

In other words, I'm working together with PC inside a for loop and want it to pause after several iterations and wait for me.

Pseudocode.

for f in /some/path/*.txt ; do
  echo unpacking $f
  ./prepare $f
if human wants to review ; then
     human is reviewing then cleans, and PC waits
  fi
done

The reason for such weird procedure is that preprocessing of one "source" .txt file creates several dozens other files, I need to view all of them and select a few samples (usually 1-2), suitable to train a network.

I could run for f in /some/path/*.txt ; do ./prepare $f ; done but this command would create several hundreds of files, and this amount overwhelms.

Answer 1

Glob qualifiers

Glob qualifiers can replace most uses of ls or find to enumerate files. They're a unique feature of zsh.

For example, $(ls /some/path/*.txt | head -2) (enumerate files in lexicographic order, keep only the first two files) is equivalent¹ to /some/path/*.txt(N[1,2]) in zsh. The N qualifier ensures that the list is empty if there are no matches, and the [from,to] qualifier limits the matches to the specified range.

Without the N qualifier, under default options, your script will exit with an error message if there are no matching files.

You can use the o or O qualifier to control the order of the files. For example, /some/path/*.txt(Nom[1,2]) takes the two newest files.

¹ _{There are some slight differences, generally at zsh's benefit. Using ls tends to be problematic with file names containing special character such as spaces or newlines or invalid byte sequences, whereas zsh's built-in features work reliably on all file names. Error management is different in corner cases. Here, as you forgot the -d option to ls, you'd also have problems if some of those *.txt files were of type directory as ls would list their contents instead.}

---

I don't see how taking two files helps with your overall objective, though. If you want to have a way to process all files, but allow a human to review the first few, you could show a step/continue/abort prompt. Something like this:

pause=1
for f in /some/path/*.txt ; do
  print -ru2 unpacking $f
  ./prepare $f
if ((pause)); then
    print -ru2 -- "$f output is ready for review."
    c=
    while [[ $c != [anq] ]]; do
      read -k1 "c?Process (N)ext, (A)ll, (Q)uit? " && c=${c:l}
    done
    echo
    case $c in
      a) pause=0;;
      q) break;;
    esac
  fi
done

Answer 2

You could use a counter with the for loop. This should work in any POSIX-compatible shell.

This is an equivalent to the first code snippet in the question.

i=0
for f in /some/path/*.txt ; do
    if [ "$((i += 1))" -gt 2 ] ; then
        break
    fi
    echo "unpacking $f"
    ./prepare.sh "$f" && rm -v "$f"
done

As written in Gilles' answer, zsh has features to implement this in a simpler way. See this answer for an explanation of the glob qualifiers.

for f in /some/path/*.txt(N[1,2]) ; do
    echo "unpacking $f"
    ./prepare.sh "$f" && rm -v "$f"
done

---

Instead of breaking the loop you can also wait for some input.

i=0
for f in /some/path/*.txt ; do
    if [ "$((i += 1))" -gt 2 ] ; then
        i=1
        printf "press Enter to continue"
        read dummy
    fi
    printf "unpacking %s\n" "$f"
    ./prepare.sh "$f" && rm -v "$f"
done

Note 1: In the conditional branch I use i=1 instead of i=0 because the script will already process the first file of the next group in this iteration.

Note 2: I put the counting and condition at the beginning of the loop instead of at the end, because at this point it is clear that there is another file to be processed. This avoids a pause after the last file.

Note 3: I edited the scripts and added quotes around $((i += 1)) because Stéphane Chazelas mentioned in a comment that in most POSIX compatible shells globbing and splitting will be performed on the result of the arithmetic expansion which can lead to unwanted effects especially when IFS contains decimal digits (which I consider an unusual case). Furthermore, if the pattern does not match any file, a loop iteration will be performed with f set to the literal file name pattern. This can result in error messages or unexpected results. To avoid this, make the loop body conditional with e.g. if [ -f "$f" ] .... If these special cases are not relevant for the intended (interactive) use, the script can be kept more simple.

Answer 3

In Bash you could shove the filenames in an array and loop over an array slice. And loop over that to repeat:

#!/bin/bash
shopt -s nullglob
files=(/some/path/*.txt)
for (( i = 0; i < "${#files[@]}"; i += 2)); do
    for f in "${files[@]:i:2}"; do
        printf "processing %s\n" "$f";
    done
    read -p "press enter to continue with the next set (or end)..."
done 

Or restructured to avoid a possible prompt at the very end, at the cost of losing simplicity:

#!/bin/bash
shopt -s nullglob
files=(/some/path/*.txt)
i=0 
while true; do
    for f in "${files[@]:i:2}"; do
        printf "processing %s\n" "$f";
    done
    (( i += 2 ))
    (( i >= "${#files[@]}" )) && break
    read -p "press enter to continue with the next set..."
done

Answer 4

For completeness, one way to process files in batches in zsh is with:

files=( /some/path/*.txt(N) )
while (( $#files )) {
  process-up-to-5-files $files[1,5]
  files[1,5]=()
}

files[1,5]=() can also be written shift 5 files but with the caveat that shift fails if $files contains fewer than 5 elements.

Answer 5

I'd suggest using gnu-parallel

1. Place your task into a script

#!/bin/bash
echo unpacking ${1}
/full/path/to/prepare.sh ${1} && rm -v ${1}

2. Run it via gnu-parallel (actual name of executable may vary depending on your distribution and how you installed the program)

parallel --halt soon,success=5 /path/to/script {} ::: *.txt

parallel by default runs one job per CPU, --halt soon,success=5 means "stop processing after 5 jobs ran successfully, but let running jobs finish". {} replaces the filename and ::: is the separator to the argument list.

It will not wait for a "continue" from your side, but since you remove the original file, you can just restart the process and have no doubles.