Feed function specific files from the directory

Question

I need to feed a program some specific files, in the correct order and grouped two by two.

If I have

A_file.txt
B_file.txt
C_file.txt
D_file.txt

I need to feed it to a program so that file A and B are processed first, then C and D and so on. In essence:

for i in *.txt; do 
   some_program A_file.txt B_file.txt > output_AB
   some_program C_file.txt D_file.txt > output_CD

I know that the above doesn't make sense, but it was to illustrate the point. Essentially, iterate over all .txt files in the folder but feed them two at a time to the program, then move to the next two.

Looking to learn, many thanks.

Answer 1

You could do this with the xargs command. If I have these files:

$ ls
A_file.txt  B_file.txt  C_file.txt  D_file.txt  E_file.txt  F_file.txt  G_file.txt  H_file.txt

Then I can process these two at a time like this:

$ find . -type f | xargs -n2 echo some_program
some_program ./A_file.txt ./B_file.txt
some_program ./C_file.txt ./D_file.txt
some_program ./E_file.txt ./F_file.txt
some_program ./G_file.txt ./H_file.txt

Here I'm simply calling echo, but you could of course drop the echo and actually run some_program instead. This will process two files at a time...but it doesn't handle generating an output filename for each invocation.

If we make it a little more elaborate, we can output to a file named after the first input filename:

find . -type f | xargs -n2 sh -c 'echo some_program $1 $2 > $1.output' --

This will produce the file A_file.txt.output for A_file.txt and B_file.txt, C_File.txt.output for the next pair, and so forth. You can get fancier with the output filename by applying various transformations; for example, to get the filename you asked for in your question, you could write:

find . -type f | xargs -n2 sh -c 'echo some_program $1 $2 > output_${1:2:1}${2:2:1}' --

This will generate output filenames output_AB, output_CD, etc.

Answer 2

#!/bin/sh

set -- *_file.txt

until [ "$#" -lt 2 ]; do
    process "$1" "$2" >"output_${1%_file.txt}${2%_file.txt}"
    shift 2
done

This sets the positional parameters to the list of filenames you are interested in, based on a filename globbing pattern matching the names in the question. It then uses a loop to iterate over this list until there are less than two names left in the list ($# is the length of the list of positional parameters).

In each iteration, the first two elements of the list, $1 and $2, are processed and then shifted off the list using shift 2.

The output from the processing is redirected to a file named output_ followed by the concatenation of the variable parts of the two filenames (whatever is before the static _file.txt string in each).

This assumes that the files are named in such a way that sorting the names in lexicographical order (which the expansion of the globbing pattern will do) results in a list of names that can be paired in the way shown in the question.

Answer 3

If switching from bash to zsh is an option, then it's just:

for i j ( *.txt(N) ) some_program -- $i $j > output_$i[1]$j[1]

(N) enables nullglob from that one glob expansion so as not to report an error if there's no match.

If there's an odd number of files, then the last run will be run with $j set to the empty string. As we leave it unquoted in argument to some_program, that will result in no corresponding argument to be passed to it. Replace with "$j" if you'd rather an empty argument be passed to it in that case.

The *.txt expansion will be in alphabetical order; you can change the order to anything you want using the o, O and/or n glob qualifiers.

For an arbitrary number of files at each iteration as opposed to just 2:

files=( *.txt(N) ) n=5
while (( $#files )) {
  some_program -- $files[1,n] > output_${(Mj[])files[1,n]#?}
  files[1,5]=()
}

Or using zargs:

autoload -Uz zargs
process() some_program -- $@ > output_${(Mj[])@#?}
zargs -rl5 -- *.txt(N) -- process

In ${(Mj[])array#?}, ${array#?} would strip the leading character from each element of the array, but with M, what is Matched is returned instead. The result is joined with nothing ([]), so you get a string made of the first character of each element.

Answer 4

Dump list of files into an array and read from it.

#!/bin/bash
arr=( *.txt )
i=0
while [ $i -lt ${#arr[@]} ];
do
  echo ${arr[$i]} ${arr[ $[$i+1] ]}
  i=$[$i+2]
done

If you have odd number of files, the request ${arr[ $[$i+1] ]} will silently give you an empty string. It is up to you to decide what to do in this case.