1

Scenario:

$ process(){ echo "[$1] [$2] [$3]" ; } ; export -f process

$ process "x" "" "a.txt"
[x] [] [a.txt]

Here we see that the 2nd argument is empty string (expected).

$ find -name "*.txt" -print | SHELL=$(type -p bash) parallel process "x" ""
[x] [./a.txt] []
[x] [./b.txt] []
[x] [./c.txt] []

Here we see that the 2nd argument is the output of find (unexpected).

Expected output:

[x] [] [./a.txt]
[x] [] [./b.txt]
[x] [] [./c.txt]

How to fix?

Note: if the 2nd argument is changed from "" to "y", then the output of find is present as the 3rd argument (expected):

$ find -name "*.txt" -print | SHELL=$(type -p bash) parallel process "x" "y"
[x] [y] [./a.txt]
[x] [y] [./b.txt]
[x] [y] [./c.txt]

Why isn't the output of find present as the 3rd argument with ""?

UPD: It seems that the solution is \"\":

$ find -name "*.txt" -print | SHELL=$(type -p bash) parallel process "x" \"\"
[x] [] [./a.txt]
[x] [] [./b.txt]
[x] [] [./c.txt]

However, I'm not sure that this is the correct general solution. Here is the counterexample:

$ VAR="" ; find -name "*.txt" -print | SHELL=$(type -p bash) parallel process "x" "$VAR"
[x] [./a.txt] []
[x] [./b.txt] []
[x] [./c.txt] []
pmor
  • 599
  • 1
    So it runs the command through a shell, interpreting quotes and whitespace. Similarly if you do echo x | parallel process "foo bar" "", you'll see the output [foo] [bar] [x], instead of [foo bar] [x] – ilkkachu Feb 29 '24 at 10:30

1 Answers1

3

So, parallel runs the command through a shell, instead of executing it directly. Well, it has to, since otherwise the shell function you're using wouldn't work.

It also means that arguments with whitespace will get split:

$ echo x | parallel process "foo bar" ""
[foo] [bar] [x]

And it doesn't really matter if you quote the individual args or the whole command:

$ echo x | parallel "process foo bar"
[foo] [bar] [x]

And you can do things like this:

$ echo x | parallel process '$(date +%F)'
[2024-02-29] [x] []
$ echo x | parallel "process foo bar > test.out"
$ cat test.out
[foo] [bar] [x]

To pass arbitrary values, you'd need to get them quoted for the shell. In Bash, you could use the ${var@Q} expansion for variables:

$ var="foo bar"; echo x | parallel process "${var@Q}"
[foo bar] [x] []

And parallel looks to have an option to do just this:

--quote -q Quote command. If your command contains special characters that should not be interpreted by the shell (e.g. ; \ | *), use --quote to escape these. The command must be a simple command (see man bash) without redirections and without variable assignments.

See the section QUOTING. Most people will not need this. Quoting is disabled by default.

$ var="foo bar"; echo x | parallel --quote process "$var"
[foo bar] [x] []

Of course that will also trash redirections and such:

$ var="foo bar"; echo x | parallel --quote process "$var" ">test.out"
[foo bar] [>test.out] [x]

Of course, it will quote whitespace, so if you try to pass the command arguments as a single string, it will fail.

Note that when you do this

$ VAR="" ; ... parallel process "x" "$VAR"

the variable still contains just the empty string, which is what gets passed to parallel as an argument. To make it the same as parallel process "x" \"\", you'd need to have hard quotes in the variable, i.e. VAR=\"\", or VAR='""' or equivalent. Or use something like parallel process "x" "'$VAR'" instead. And remember that you can't blindly wrap things in quotes if the variable itself can also contain quotes. This will fail:

$ var="ain't so"; echo x | parallel process "'$var'"
/usr/bin/bash: -c: line 1: unexpected EOF while looking for matching `''
/usr/bin/bash: -c: line 2: syntax error: unexpected end of file
ilkkachu
  • 138,973