Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

GNU parallel is a command line utility to run programs in parallel, on a single (usually multiprocessor) machine or over multiple machines via .

For questions about running programs in parallel in general, see . For questions about the tool called parallel from Joey Hess's moreutils, see .

External links

266 questions
21
votes
3 answers

Can GNU parallel output stdout before the program has exited?

echo 'echo "hello, world!";sleep 3;' | parallel This command does not output anything until it has completed. Parallel's man page claims: GNU parallel makes sure output from the commands is the same output as you would get had you run the commands…
Luc
  • 3,610
11
votes
1 answer

Does "parallel --jobs 10" mean that exactly 10 jobs will run?

When specifying the option --jobs to GNU parallel, what exactly does it mean? I execute: parallel --jobs 10 ./program ::: {1..100} where program is an intensive task, and the jobs are completely independent of each other. {1..100} represents…
a06e
  • 1,727
7
votes
3 answers

Using parallel on Ubuntu

I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without any action being taken. e.g. I was trying to…
Farhat
  • 297
5
votes
2 answers

GNU parallel ssh jobs: What happens to an incomplete job if the server dies?

Suppose we want to dispatch jobs to a collection of servers using GNU parallel. What would happen if one of the servers die(power failure, thermal shutdown...) while busy executing a job? Will GNU parallel just dispatch the same job to another…
niobe
  • 205
4
votes
2 answers

Using GNU parallel on line based output without temporary files

The default output mode of GNU parallel is --group: The output of each job is written to a temporary file and passed to the output of parallel only after the job has finished. When using this default output mode on data larger than the /tmp space…
Juergen
  • 674
4
votes
1 answer

Why is parallel --nice not setting the niceness?

If I run a command with nice, then I can see its process having the expected niceness value: In one terminal: nice sleep 17 and in another one: $ ps -aoni,comm | grep sleep 10 sleep But trying to do the same with GNU parallel (version 20161222,…
Charles Plessy
  • 143
3
votes
2 answers

How to print GNU Parallel command & output together?

I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd need some way to print either the git command run…
l0b0
  • 51,350
3
votes
1 answer

Use GNU parallel to run script with different input arguments

I have a script called step1.sh, and I pass one input parameter to the script ($1). I can run the script like this if I want the input to be 300.1 ./step1.sh 300.1 I want to run this script 200 times, with 200 different inputs (they are random…
adann
  • 31
3
votes
2 answers

--skip-first-line in GNU parallel not working with --pipepart?

Suppose I have a file like this: COLUMN 1 2 3 4 If I wanna run and process it with GNU parallel but skipping first line aka header, I tried this: parallel -a test.txt -k --pipepart --will-cite --skip-first-line cat However, --skip-first-line is…
user1261558
  • 241
3
votes
3 answers

Can GNU Parallel be made to output the command line executed when run in linewise mode?

Suppose I have a list of commands in file cmd_file. I run these commands via: cat cmd_file | parallel -k -I {} "{}" One of the commands fails. All of the commands use the exact same CLI tool with different inputs. Right now, I have to run across…
Chris
  • 961
  • 7
  • 20
3
votes
1 answer

gnu parallel: how to set the limit per second

I need to use parallel and set the rate limit per second, because I need to query an API that has a "5 per second" rate limit. Do I must combine -n5 and --timeout 1? Thank you
aborruso
  • 2,855
3
votes
1 answer

Is there a way to tell GNU parallel to hold off spawning new jobs until all jobs in a batch has finished?

I want to run four processes in parallel, but not spawn any new jobs until all of these four have finished. EDIT: My command looks like this: find . -name "*.log" | parallel -j 4 './process.sh {}'
user2724383
  • 33
  • 5
3
votes
3 answers

On which host does this parallel command hang?

I run this parallel command: parallel -q -j0 ssh {} 'echo {}; tmp/myscript' ::: host1 host2 host3 .... Above line shows a result for most hosts. Unfortunately the parallel command hangs on some host. The script seems to be in an endless loop. How…
guettli
  • 1,389
3
votes
3 answers

Can I concurrently pull my git repos without gnu parallel?

On my home machine, I use the script gitpull.sh to concurrently pull all the changes to the git repos under a given directory. #!/usr/bin/bash find $1 -name ".git" | sed -r 's|/[^/]+$||' | parallel git -C {} pull origin master My problem is that…
Brian Fitzpatrick
  • 2,845
2
votes
1 answer

Can I use GNU parallel to group command arguments by a column value?

Consider the data from the GNU parallel manual's example for --group-by: cat > table.csv <<"EOF" UserID, Consumption 123, 1 123, 2 12-3, 1 221, 3 221, 1 2/21, 5 EOF Is there a way to group records by one column and write all the…
Iain Samuel McLean Elder
  • 324
1
2 3 4