1

Is it possible to start GNU Parallel in the background, or somehow, send it to the background?

I want to do this so that I can use two commands that run parallel at the same time.

And why don't I execute everything in one parallel command? Two reasons:

  1. parallel slows down after a while,
  2. because I want to separate it so I can stop execution of one group without stopping everything
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Dominique
  • 5,235
  • Can you elaborate on why you'd want to do this? This sounds like an XY Problem. – slm Apr 10 '14 at 13:14
  • I have a script that downloads files with wget, I want to download as fast as possible. This is what I want. – Dominique Apr 10 '14 at 15:00
  • Understand, but you're limited by your network connection so running many wget's in parallel will not work, all you'll be doing is dividing up the fixed amount of bandwidth over many vs. 1 or 2. Rather then bother with parallel here you should just run wget backgrounded directly. Running 1-3 will likely saturate your network connection. – slm Apr 10 '14 at 15:12
  • As an a for example, see this SO Q&A titled: multiple wget -r a site simultaneously? – slm Apr 10 '14 at 15:14
  • A tool such as axel might be a better fit here too if you're downloading a file, but want to download it over multiple connections. – slm Apr 10 '14 at 15:17
  • Also, depending on the structure of the URLs you're downloading from, these approaches might be useful: http://stackoverflow.com/questions/7577615/parallel-wget-in-bash – slm Apr 10 '14 at 15:18
  • I would like to run wget separate, the thing is that the bash script does a number of operations to determine the correct URL to send to wget. I wish I could write some sort of implementation in C++ to do all the operations and have the program send the URLs to a fixed number of wgets. I've been reading a bit on Swift to do this but I'm no master programmer unfortunately. – Dominique Apr 10 '14 at 16:56

2 Answers2

1

It is possible to run parallel in background. Recently, I was working on similar scripts which I needed to execute in parallel and I got the suggestions in this answer. If you go to the link specified as part of my answer, I see the below answer.

GNU Parallel can work as a counting semaphore. This is slower and less efficient than its normal mode.

An alias for parallel --semaphore is sem. The default is to allow only one program to run at a time (technically called a mutex). The program is started in the background. Use --wait for all 'sem's to finish:

sem 'sleep 1; echo The first finished' &&
    echo The first is now running in the background &&
    sem 'sleep 1; echo The second finished' &&
    echo The second is now running in the background
  sem --wait

If you see the examples in the link that I had provided, you can definitely find what you are looking for.

Community
  • 1
Ramesh
  • 39,297
  • My goal with starting parallel in background is to be able to execute even more things at the same time. – Dominique Apr 10 '14 at 00:56
  • I understand it :) I would also go through Amdahl's law to understand how much parallelism can be effectively achieved. – Ramesh Apr 10 '14 at 01:00
1

I often run things several different groups of task in parallel:

parallel do_stuff ::: x y z

I then realize that do_stuff takes a long time to run, and that I want my terminal back, so I press CTRL-Z and type:

bg

to put the job in the background.

Then I might start another job:

parallel do_other_stuff ::: foo bar baz

This new job will run in parallel with do_stuff.

Ole Tange
  • 35,514