How to use defined function with xargs

Question

This is my code

#!/bin/bash

showword() {
  echo $1
}

echo This is a sample message | xargs -d' ' -t -n1 -P2 showword

So I have a function showword which echoes whatever string you pass as a parameter to the function.

Then I have xargs trying to call the function and pass one word at a time to the function, and run 2 copies of the function in parallel. The thing that is not working is xargs doesn't recognize the function. How can I achieve what I am trying to do, how can I make xargs work with the function showword?

Also discussed in https://stackoverflow.com/questions/11003418/calling-shell-functions-with-xargs — Joshua Goldberg, Feb 28 '19 at 17:38

score 52 · Accepted Answer · edited Oct 15 '22 at 17:07

52

Try exporting the function, then calling it in a subshell:

showword() {
  echo $1
}
export -f showword
echo This is a sample message | xargs -d' ' -t -n1 -P2 bash -c 'showword "$@"' _

This causes xargs to execute

bash -c 'showword "$@"' _ This
bash -c 'showword "$@"' _ is
bash -c 'showword "$@"' _ a
            ︙

The arguments passed to the bash command are, well, passed into the bash environment, but starting from 0. So, inside the function,

$0 is “_” and $1 is “This”
$0 is “_” and $1 is “is”
$0 is “_” and $1 is “a”
︙

See Bash -c with positional parameters.

Note that export -f works only in Bash, and -Pn (--max-procs=max-procs) works only in GNU xargs.

edited Oct 15 '22 at 17:07

G-Man Says 'Reinstate Monica'

22,870

answered Oct 01 '14 at 06:13

cuonglm

153,898

7

One more question, why is the trailing _ required? Is it to stop xargs from processing anything else? – GMaster Oct 01 '14 at 06:51
4

@FazleA.: read http://unix.stackexchange.com/questions/152391/bash-c-with-positional-parameters – cuonglm Oct 01 '14 at 07:08
2

I am getting "export: Illegal option -f" when I do export -f – Sagar Chamling Nov 22 '18 at 06:35
@SagarChamling export -f works in bash only – cuonglm Nov 22 '18 at 08:06
1

@cuonglm this does invoke a new bash process for every xargs item, right? – NarūnasK May 10 '19 at 19:42
@NarūnasK No, for each xargs invocation. – cuonglm May 11 '19 at 13:27
2

Why does this work? Or, more specifically, why is the bash -c subshell needed (why can't xargs just detect and use the function just defined in the same shell process)? – Magnus Mar 13 '21 at 22:26
1

@Magnus because xargs run the command in new process, not with the original one. That's why you need export f to make the function visible in sub-shells. – cuonglm Mar 15 '21 at 02:25

GMaster · Answer 2 · 2021-12-09T16:57:20.933

Just adding an alternative solution using parallel which I have started using in place of xargs. The task is much easier with parallel

#!/bin/bash
showword() {
  echo $1
}
export -f showword
parallel -j2 showword {} ::: This is a sample message

-j2 makes sure 2 copies of the function is run in parallel
::: anything after this is passed as separate arguments to parallel, separation is whitespace
{} is replaced by the argument passed into parallel and fed into the showword function

If you are using zsh shell this solution will not work since zsh does not have any feature to export functions. You will need something like this:

#!/usr/bin/zsh
showword() {
  echo $1
}
add the following to your .zshrc if you want env_parallel in your shell permanently
source /usr/bin/env_parallel.zsh
env_parallel -j2 --env showword showword {} ::: This is a sample message

Where does /usr/bin/env_parallel.zsh come from? I don't have it on my system... — friederbluemle, Oct 19 '22 at 21:18
@friederbluemle On Ubuntu, if I run the command dpkg -S /usr/bin/env_parallel.zsh, it says the file /usr/bin/env_parallel.zsh comes from the package parallel — GMaster, Oct 20 '22 at 02:55

score 0 · Answer 3 · edited Feb 09 '24 at 15:27

Let we have a function which takes argument and does anything (mine prints it two times)

$ twice() { echo $1$1 }
$ twice "hello"
hellohello

It doesn't work with pipe as it doesn't read any input

$ echo "hello" | twice
<nothing>

We can simply fix this by reading input to temp variable

$ echo "hello" | read s; twice $s
hellohello

Or with helper function

$ call() { read s; $1 $s }
$ echo "hello" | call twice
hellohello

How to use defined function with xargs

3 Answers3

add the following to your .zshrc if you want env_parallel in your shell permanently

Linked