Pipe globs to ls

Question

Contents of file filelist:

/some/path/*.txt
/other/path/*.dat
/third/path/example.doc

I want to list those files, so I do:

cat filelist | xargs ls

But instead of expanding those globs, I get:

ls: cannot access '/some/path/*.txt': No such file or directory  
ls: cannot access '/other/path/*.dat': No such file or directory  
/third/path/example.doc

Answer 1

Shells expand globs. Here, that's one of the very rare cases where the implicit split+glob operator invoked upon unquoted command substitution in Bourne-like shells other than zsh can be useful:

IFS='
' # split on newline only
set +o noglob # make sure globbing is not disabled
ls -ld -- $(cat filelist) # split+glob

In zsh, you'd do:

ls -ld -- ${(f)~"$(<filelist)"}

Where f is the parameter expansion flag to split on linefeeds, and ~ requests globbing which is otherwise not done by default upon parameter expansion nor command substitution.

Note that if the list of matching files is large, you can run into an Argument list too long error (a limitation of the execve() system call on most systems), which xargs would have otherwise worked around. In zsh, you can use zargs instead:

autoload zargs
zargs --eof= -- ${(f)~"$(<filelist)"} '' ls -ld --

Where zargs will split the list and run ls several times to avoid the limit as necessary as xargs would.

Or you could pass the list to a command that is builtin (so doesn't involve the execve() system call):

To just print the list of files:

print -rC1 -- ${(f)~"$(<filelist)"}

Or to feed it to xargs NUL-delimited:

print -rNC1 -- ${(f)~"$(<filelist)"} |
  xargs -r0 ls -ld --

Note that if any of the globs fails to match a file, in zsh, you'll get an error. If you'd rather those globs to expand to nothing, you'd add the N glob qualifier to the globs (which enables nullglob on a per-glob basis):

print -rNC1 -- ${(f)^~"$(<filelist)"}(N) |
  xargs -r0 ls -ld --

Adding that (N) would also turn all the lines without glob operators into globs allowing to filter out files referenced by path and that don't exist; it would however mean you can't use glob qualifiers in the globs in filelist unless you express them as (#q...) and enable the extendedglob option. Also beware that as qualifiers can run arbitrary code, it's important the contents of the filelist file comes from a trusted source.

In other Bourne-like shells, including bash, globs that don't match are left as-is, so would be passed literally to ls which would likely report an error that the corresponding file doesn't exist.

In bash, you could use the nullglob option (which it copied from zsh) and handle the case where none of the globs match specially:

shopt -s nullglob
IFS=$'\n'
set +o noglob
set -- $(<filelist)
(( $# == 0 )) || printf '%s\0' "$@" | xargs -r0 ls -ld --

bash, doesn't have any equivalent for zsh's glob qualifiers. To make sure lines without glob operators (such as your /third/path/example.doc) are treated as globs and removed if they don't correspond to an actual file, you could add @() to the lines (requires extglob). That won't work however for line that end in / characters. You could however add @() to the last non-/ character and rely on the fact that / always exists

shopt -s nullglob extglob
IFS=$'\n'
set +o noglob
set -- $(LC_ALL=C sed 's|.*[^/]|&@()|' filelist)
(( $# == 0 )) || printf '%s\0' "$@" | xargs -r0 ls -ld --

In any case, note that the list of supported glob operators vary greatly with the shell. The only one you're using in your sample (*) should be supported by all though.

Answer 2

You can modify your script a little, and call sh from xargs:

cat filelist | xargs -i -- /bin/sh -c 'ls $1' _X_ {}

Or let xargs read the file itself:

xargs -a filelist -i -- /bin/sh -c 'ls $1' _X_ {}

Answer 3

while read filepattern
do
    ls $filepattern
done < filelist

In this case you want the variable value to be expanded by file globbing, so you have to use the variable without quotes.

The shell will replace /some/path/*.txt with the list of matching files.

If you only want to list the file names you don't need ls. You can use echo instead like echo $filepattern.

In contrast to the code above, with quotes you will get the same errors as in the example from the question because the shell will pass the unchanged strings, e.g. /some/path/*.txt to ls.

while read filepattern
do
    ls "$filepattern"
done < filelist