Can I Indirect the argument list to a bash script from a text file?

Question

I'm thinking of something like ls @files.lst doing what ls [contents of files.lst] would do, globbing included.

I think I'm remembering the above syntax, (at-sign, filename) from somewhere (maybe an old Digital OS from sometime before the last ice age?). Searching for permutations of Bash, indirect, arguments, list, ... isn't getting me anywhere. I know there are ways to write this into the (every?!) script, but does bash already implement it in some more general way that so has far eluded me?

Update: This is for BSD/Darwin (ATM) but I appreciate the more general answers, too. I'm on a Mac, if that's not now obvious, so computational inefficiency won't hamper anyone else and the workload is pretty small anyway. Of more interest is the ability to use existing shell scripts that were written to receive their commands from the command line, and most of the time, they will.

Answer 1

If you want to do something for each line of a text file, there are generally two ways of doing it.

1. Read the file line by line and do whatever you need to do to the line in the body of the loop:

   while IFS= read -r pathname; do
       ls -d -- "$pathname"
   done <files.list
   

   The above reads lines from the file files.list and executes ls -d -- on each.

   Related:

   • Understanding "IFS= read -r line"
   • Why is using a shell loop to process text considered bad practice?

2. Use a utility that makes the looping implicit for you, like xargs or GNU parallel:

   xargs -I {} -- ls -d -- {} <files.list
   

   The code does the same as the loop above but uses xargs to execute ls -d -- for each argument read from files.list. An argument is a line where newlines may be embedded if they are escaped (quotes are also processed, so they must be escaped to be retained).

   xargs -- ls -d -- <files.list
   

   The above would run ls -d -- on as many of the arguments in the file as possible at once. One needs to quote arguments containing embedded tabs or spaces and escape individual embedded newlines and quotes.

   The GNU parallel utility is more generic than xargs and allows for more control over the executed jobs.

3. Since the shell is often the slowest language you can pick for processing data, if you need to do something other than executing a command for each line in a file, then it's probably better to do so in a language that makes it easier to do this. Examples of such languages:

   • Awk
   • Perl
   • Python

In either case, since you store newline-delimited pathnames in your file, you are disqualified from using pathnames containing newlines (these are legal on Unix systems) unless quoted adequately for use with xargs (see the xargs manual).

Answer 2

Learning details of Bash; www.tldp.org -> Bash guides

$ man bash tells some details, but in terse form.

Now bypassing those sources above that has most pf the details;
a very simplistic and incomplete answer:

$ ls -l $(cat file)
... will read file and insert that text as arguments for ls, replacing LF (\n) characters with space (SP) characters.

Note though that globbing is unlikely to happen unless you do tricks here.

Note another problem; if any of the lines in the file contains a single space or more, the words on that line will become separate arguments to ls.

Whether a specific command (bash specific or not) can take arguments from within a file varies;
man command_name generally will tell details, for a specific command named command_name (replace that with a true command name).

Answer 3

Read the output of the man xargs command, and do something like:

xargs -r <files.lst somecommand

xargs will fill somecommand's command line with as many items from files.lst as will fit, and will use as many somecommand executions as needed. See xargs --show-limits </dev/null.

You can also use xargs to feed the filenames to somecommand 1 at a time:

xargs -r -n 1 <files.lst somecommand

Answer 4

In the zsh shell:

cmd -- $(<file)

With the default value of $IFS, would pass each space/TAB/NL/NUL delimited word in file to cmd (where $(<file) (from ksh) expands to the contents of file without the trailing newline characters, and when left unquoted is subject to IFS-splitting¹).

cmd -- ${(f)"$(<file)"}

Would pass the contents of each non-empty line as separate arguments to cmd (where the f parameter expansion flag splits on newlines aka linefeeds, and empty arguments are discarded as the expansion is not quoted).

Replace f with 0 to split on NULs instead of NLs.

cmd -- "${(Q@)${(zZ[n])$(<file)}}"

Would interpret zsh-style quoting in the file, with the z flag doing the quote-aware tokenisation, Z[n] tuning it so that newline is just considered as a separator, Q removes the quotes and @ combined with quoting preserve empty elements.

For example, on a file like:

'' "foo bar"
whatever $'\n'

and printf '<%s>\n' as the command, that would give:

$ printf '<%s>\n' $(<file)
<''>
<"foo>
<bar">
<whatever>
<$'\n'>

$ printf '<%s>\n' ${(f)"$(<file)"}
<'' "foo bar">
<whatever $'\n'>

$ printf '<%s>\n' "${(Q@)${(zZ[n])$(<file)}}"
<>
<foo bar>
<whatever>
<
>

---

^{¹ in the Korn shell (or bash), it would subject to IFS-splitting and globbing, which you generally don't want unless those words happen to be meant to be glob patterns. Also note that ksh/bash don't split on NULs by default (and generally can't cope with those).}