The best way to expand glob pattern?

Question

I need to expand a glob pattern (like ../smth*/*, or /etc/cron*/) into a list of files, programmatically. What would be the best way to do it?

Answer 1

Just let it expand inside an array declaration's right side:

list=(../smth*/)          # grab the list
echo "${#list[@]}"        # print array length
echo "${list[@]}"         # print array elements
for file in "${list[@]}"; do echo "$file"; done  # loop over the array

Note that the shell option nullglob needs to be set.
It is not set by default.
It causes globs with no match to expand to nothing instead of causing an error (in zsh or bash -O failglob) or be passed literally (all other Bourne-like shells).

Set it in bash with

shopt -s nullglob

or in zsh or yash with

set -o nullglob

though in zsh (where the nullglob initially came from), you'd rather use the (N) glob qualifier to avoid having to change a global setting:

list( ../smth*/(N) )

The ksh93 equivalent:

list=( ~(N)../smth*/ )

Answer 2

compgen is a Bash built-in that you can pass an escaped(!) pattern to, and it outputs matches, returning true or false based on whether there were any. This is especially useful if you need to pass the glob pattern from a variable/script argument.

glob_pattern='../smth*/*'
while read -r file; do
    # your thing
    echo "read $file"
done < <(compgen -G "$glob_pattern" || true)

adding the || true prevents a false return from compgen causing any problems. This method avoids issues with no matches and does not require changing nullglob options.

If you need the items in an array, just initialise one with files=() before the loop, and files+=("$file") inside the loop. You can then see if there were any matches by simply checking the length of the array with if [[ ${#files[@]} -gt 0 ]]; then.

Answer 3

I wanted to use a standard input (pipe) in case a resulting command exceeds a command line length limit. The following command worked for me:

echo "../smth*/*" "/etc/cron*/" | xargs -n1 -I{} bash -O nullglob -c "echo {}" | xargs -n1

or for a list of globs:

cat huge_glob_list.txt | xargs -n1 -I{} bash -O nullglob -c "echo {}" | xargs -n1

Answer 4

No need to overcomplicate things:

echo your/stuff*

Answer 5

Recently I have the same question. And I find that the solution is very simple: (and it is POSIX compliant.)

• Set $IFS to empty string, which disable word splitting by whitespace characters.
• Then just unquote the variable to let it expands the globs.

Example code illustrated in for-loop:

pattern='some * dir/my file *'
unset old_IFS ; [ -n "${IFS+x}" ] && old_IFS=${IFS} ; IFS=''
IFS=''
for f in ${pattern} ; do
    IFS=${old_IFS} ; [ -z "${old_IFS+x}" ] && unset IFS
printf 'Filenames: %s \n' "${f}"

done

Please note that I do not set nullglob by shopt -s nullglob as shopt is not defined in POSIX. If the glob pattern is not found, the pattern expands to itself. Filenames: some dir/my file * is printed in the above code. It is easy to add an if [ -e "${f}" ]; then ... check if necessary.

The same approach can be used to set the positional parameters also.

pattern='some * dir/my file *'
unset old_IFS ; [ -n "${IFS+x}" ] && old_IFS=${IFS} ; IFS=''
set -- ${pattern}
IFS=${old_IFS} ; [ -z "${old_IFS+x}" ] && unset IFS
unset old_IFS
printf '[%s]\n' "$@"

Note that we cannot make it into one-liner IFS='' command set -- ${pattern}. This one-liner does not disable word splitting.

It may be used in function parameters, but it is not recommended. The restore of $IFS has to be located at the first statement of the function, which is not symmetric in style and easily be forgotten.

func() {
    IFS=${old_IFS} ; [ -z "${old_IFS+x}" ] && unset IFS
    unset old_IFS
    printf '[%s]\n' "$@"
}
pattern='some * dir/my file *'
unset old_IFS ; [ -n "${IFS+x}" ] && old_IFS=${IFS} ; IFS=''
func $pattern

Personally I would prefer passing $pattern into the function, then set -- $pattern inside the function. But it is not always possible if the function carries other positional parameters also.

func() {
    unset old_IFS ; [ -n "${IFS+x}" ] && old_IFS=${IFS} ; IFS=''
    set -- ${pattern}
    IFS=${old_IFS} ; [ -z "${old_IFS+x}" ] && unset IFS
    unset old_IFS
printf '[%s]\n' "$@"

}
pattern='some * dir/my file *'
func $pattern

This approach works for both the pattern and the filepaths:

• if they contain whitespace characters, indeed all characters, and
• if they contain glob characters, use escape \* to match a literal *, and
• when the glob characters locates in the directory path components and/or in the filename components. (If glob in directory path is necessary, it cannot be easily implemented using find.)

Answer 6

Nowadays most linux distributions have python included, so you can just run the following command in shell

python -c 'from glob import glob; print(glob("*"))'

You are free modify the python script to meet your requirement, for example, dump to json format string.

python -c 'from glob import glob; from json import dumps; print(dumps(glob("*")))'