Is there a way to tell GNU parallel to hold off spawning new jobs until all jobs in a batch has finished?

Question

I want to run four processes in parallel, but not spawn any new jobs until all of these four have finished.

EDIT: My command looks like this:

find . -name "*.log" | parallel -j 4 './process.sh {}'

Answer 1

That very question has recently been put on the mailing list: https://lists.gnu.org/archive/html/parallel/2019-06/msg00003.html

The short answer is: As GNU Parallel is now, it cannot do this. A (crappy) workaround is here:

https://lists.gnu.org/archive/html/parallel/2019-06/msg00008.html

in-line:

#!/bin/bash

parpar() {
    . `which env_parallel.bash`
    env_parallel --session

    inner() {
        parallel "${command[@]}"
    }
    export -f inner

    command=()
    while [[ $# -gt 0 ]]; do
        if [[ $1 == ",,," ]] ; then
            break
        fi
        command+=("$1")
        shift
    done
    printf "%s\n" "$@" |
        env_parallel --pipe --recend ',,,\n' --rrs -j1 -N1 inner
}

# Example (,,, = wait for completion)
parpar -v sleep {}\; echo {} ,,, 3.9 4 4.1 ,,, 1 2 3 ,,, 0.2 0.3 0.1