xargs with stdin/stdout redirection

Question

I would like to run:

./a.out < x.dat > x.ans

for each *.dat file in the directory A.

Sure, it could be done by bash/python/whatsoever script, but I like to write sexy one-liner. All I could reach is (still without any stdout):

ls A/*.dat | xargs -I file -a file ./a.out

But -a in xargs doesn't understand replace-str 'file'.

Thank you for help.

Answer 1

First of all, do not use ls output as a file list. Use shell expansion or find. See below for potential consequences of ls+xargs misuse and an example of proper xargs usage.

1. Simple way: for loop

If you want to process just the files under A/, then a simple for loop should be enough:

for file in A/*.dat; do ./a.out < "$file" > "${file%.dat}.ans"; done

---

2._pre1 Why not   ls | xargs ?

Here's an example of how bad things may turn if you use ls with xargs for the job. Consider a following scenario:

• first, let's create some empty files:

  $ touch A/mypreciousfile.dat\ with\ junk\ at\ the\ end.dat
  $ touch A/mypreciousfile.dat
  $ touch A/mypreciousfile.dat.ans
  

• see the files and that they contain nothing:

  $ ls -1 A/
  mypreciousfile.dat
  mypreciousfile.dat with junk at the end.dat
  mypreciousfile.dat.ans
  
  $ cat A/*
  

• run a magic command using xargs:

  $ ls A/*.dat | xargs -I file sh -c "echo TRICKED > file.ans"
  

• the result:

  $ cat A/mypreciousfile.dat
  TRICKED with junk at the end.dat.ans
  
  $ cat A/mypreciousfile.dat.ans
  TRICKED
  

So you've just managed to overwrite both mypreciousfile.dat and mypreciousfile.dat.ans. If there were any content in those files, it'd have been erased.

---

2. Using  xargs : the proper way with  find 

If you'd like to insist on using xargs, use -0 (null-terminated names) :

find A/ -name "*.dat" -type f -print0 | xargs -0 -I file sh -c './a.out < "file" > "file.ans"'

Notice two things:

1. this way you'll create files with .dat.ans ending;
2. this will break if some file name contains a quote sign (").

Both issues can be solved by different way of shell invocation:

find A/ -name "*.dat" -type f -print0 | xargs -0 -L 1 bash -c './a.out < "$0" > "${0%dat}ans"'

---

3. All done within find ... -exec

 find A/ -name "*.dat" -type f -exec sh -c './a.out < "{}" > "{}.ans"' \;

This, again, produces .dat.ans files and will break if file names contain ". To go about that, use bash and change the way it is invoked:

 find A/ -name "*.dat" -type f -exec bash -c './a.out < "$0" > "${0%dat}ans"' {} \;

Answer 2

Use GNU Parallel:

parallel ./a.out "<{} >{.}.ans" ::: A/*.dat

Added bonus: You get the processing done in parallel.

Watch the intro videos to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 3

Try doing something like this (syntax may vary a bit depending on the shell you use):

$ for i in $(find A/ -name \*.dat); do ./a.out < ${i} > ${i%.dat}.ans; done

Answer 4

For simple patterns, the for loop is appropriate:

for file in A/*.dat; do
    ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

For more complex cases where you need another utility to list the files (zsh or bash 4 have powerful enough patterns that you rarely need find, but if you want to stay within POSIX shell or use fast shell like dash, you will need find for anything non-trivial), while read is most appropriate:

find A -name '*.dat' -print | while IFS= read -r file; do
   ./a.out < "${file}" > "${file%.dat}.ans" # Never forget the QUOTES!
done

This will handle spaces, because read is (by default) line-oriented. It will not handle newlines and it will not handle backslashes, because by default it interprets escape sequences (that actually allows you to pass in a newline, but find can't generate that format). Many shells have -0 option to read so in those you can handle all characters, but unfortunately it's not POSIX.

Answer 5

There's no need to complicate it. You could do it with a for loop:

for file in A/*.dat; do
  ./a.out < "$file" >"${file%.dat}.ans"
done

the ${file%.dat}.ans bit will remove the .dat filename suffix from the filename in $file and instead add .ans to the end.

Answer 6

I think you need at least a shell invocation in the xargs:

ls A/*.dat | xargs -I file sh -c "./a.out < file > file.ans"

Edit: It should be noted that this approach does not work when the filenames contain whitespace. Can't work. Even if you used find -0 and xargs -0 to make xargs understand the spaces correctly, the -c shell call would croak on them. However, the OP explicitely asked for an xargs solution, and this is the best xargs solution I came up with. If whitespace in filenames might be an issue, use find -exec or a shell loop.