2

I am a super beginner with bash and coding in general so bear with me. I have a list of files (>1000) and I need to convert them to another format. However I need to run the command on 40 files a time.

Here is what I tried (but it basically run the command as there was nothing before, the xargs is not working).

path=/home/dir1/dir2/dir3;
ls ${path} >> ${path}/LIST;
FILES=${path}/LIST;
xargs -n 40 <<$FILES | xargs commandname
Archemar
  • 31,554
Adone
  • 21

4 Answers4

4

Your issue is on reading your list:

FILES=${path}/LIST;

should be

FILES=$(<"${path}/LIST")

However,

  • You could also let xargs read the file directly: xargs -a "${path}/LIST".
  • You don't need duplicate xargs at all.
  • You should always double quote your filename variables!
  • Your script has a lot of issues if your paths contain spaces or newlines.
  • Add -r to xargs to prevent running your command without arguments if no files found.

Anyways,

Don't parse ls.

You should use an array instead.

Also, it's good habit to use \0 as delimiter together with xargs -0 to prevent issues with newlines as part of file names.

shopt -s nullglob # avoid `*` as file if no files found in $path
path=/home/dir1/dir2/dir3
files=("${path}"/*)
printf '%s\0' "${files[@]}" | xargs -0 -r -n40 commandname
shopt -u nullglob

OR use parallel instead of xargs:

shopt -s nullglob
parallel -j1 -n40 commandname ::: /home/dir1/dir2/dir3/*
pLumo
  • 22,565
3

With zsh instead of bash.

autoload zargs
zargs -rl40 -- /home/dir1/dir2/dir3/*(nN) -- commandname

(n for numericglobsort for a sorting order which is generally better for numbered files (so file10 be sorted after file2 for instanace), and N for nullglob (so it doesn't complain if /home/dir1/dir2/dir3/* does match and file when /home/dir1/dir2/dir3 doesn't contain any non-hidden file or is not readable)).

Stéphane Chazelas
  • 544,893
0

I think something like this would work, without needing to use GNU extensions.

set -- # clear positional arguments
for file in /home/dir1/dir2/dir3/*
do
    [ -e "$file" ] || continue
    set -- "$@" "$file"
    if [ "$#" -eq 40 ]
    then
        commandname "$@"
        set --
    fi
done
# in case the number of files is a number not divisible by 40
if [ "$#" -gt 0 ] && [ "$#" -lt 40 ]
then
    commandname "$@"
fi

I don't know what the command is, but when I made a function like

commandname()
{
    printf 'received %d arguments\n' "$#"
    printf 'listing received filenames\n'
    counter=0
    for par in "$@"
    do
        counter="$(( counter+1 ))"
        printf 'argument number %d: %s' "$counter" "$par" | LC_ALL=POSIX tr -d '[:cntrl:]'
        printf '\n'              
    done
}

it showed a proper number of them received, and listed all of them correctly, as far as I have seen. I tested that on most of the filenames listed here How can I test my shell script's file-handling robustness? (53 files).

It's also worth noting that this won't find hidden files (ones that start with a dot), but what you wrote in OP doesn't either, so I assume that's fine. If you want only regular files, change -e "$file" ] to [ -f "$file" ].

udongest
  • 1
0
find /home/dir1/dir2/dir3 -maxdepth 1 -print0 |
  xargs -0 -n 40 -P 0 commandname

Note that if the number of files is not a multiple of 40, one of the commandname process would not run with 40 arguments.

Weihang Jian
  • 1,227