1

Similar to this thread, I have a remote machine with 8 cores that I want to use for running scripts in parallel (1 script per core at a time).

However, I don't have multiple bash scripts but a single Python3 script that I want to run with different inputs. I tried parallel python3 -c main.py input*, parallel -j 100% python3 -c main.py ::: input*, and parallel python3 main.py input* but nothing worked.

The exact error message is:

parallel: Error: -g has been retired. Use --group.
parallel: Error: -B has been retired. Use --bf.
parallel: Error: -T has been retired. Use --tty.
parallel: Error: -U has been retired. Use --er.
parallel: Error: -W has been retired. Use --wd.
parallel: Error: -Y has been retired. Use --shebang.
parallel: Error: -H has been retired. Use --halt.
parallel: Error: --tollef has been retired. Use -u -q --arg-sep -- and --load for -l.

I don't understand how this is related to my input. I didn't use any of these options.

I'm fairly new and inexperienced with Unix and couldn't get it to work myself or with googling. Any help is appreciated. Do I have to write a shell-script to help me with that?

Community
  • 1
stefanbschneider
  • 141
  • 6
  • There are a lot of different versions of parallel out there, do you know which one you have? (parallel --version or man parallel should list it somewhere) – Wieland Dec 01 '16 at 09:55
  • It's GNU parallel 20141022 – stefanbschneider Dec 01 '16 at 10:12
  • Make sure you do not have a variable PARALLEL set in your environment. Check with echo $PARALLEL. Try preceding your command with PARALLEL=, as in PARALLEL= parallel python3 ... all on one line. – meuh Dec 01 '16 at 18:05
  • Good idea but unfortunately it's not the problem. No environmental variable PARALLEL is set and preceding PARALLEL= leads to the same error message. Can I somehow reinstall parallel? Does it even make sense? – stefanbschneider Dec 01 '16 at 18:59

3 Answers3

2

The problem was actually with how parallel was installed on the remote machine (running the newest Ubuntu). I came across a thread solving my problem: Run sudo rm /etc/parallel/config after installation on Ubuntu to get rid of the config which caused my error messages.

The command I use to run my python script with different inputs in parallel is: parallel -j 100% python3 main.py ::: inputs*

Nevertheless, thanks to everyone who was helping!

Community
  • 1
stefanbschneider
  • 141
  • 6
0

You can do that by assigning every script to specific core using taskset first you need to get the process id for your script with ps aux | grep <scriptname, input> or using pgrep -f <scriptname, input> then pass it to taskset like this taskset -pc 0 $pid, this will assign the process to core 1. for more info about taskset http://linuxcommand.org/man_pages/taskset1.html

Wissam Roujoulah
  • 3,229
  • So I can't just use parallel and let it handle the assignment to the different cores? – stefanbschneider Dec 01 '16 at 09:50
  • i am not aware with parallel tool but the way that i mentioned will do the job but we will need to write small script to make this process auto – Wissam Roujoulah Dec 01 '16 at 09:53
  • It sounds like I have to manually assign the different jobs to different cores. And what happens if I assign multiple jobs to the same core? Will it run them one after another or will run it concurrently by switching between the threads (I don't want that)? – stefanbschneider Dec 01 '16 at 10:14
  • don't run two scripts in one core at the same time wait until it finish then start the other one, take a look into this it may help you with ordering http://unix.stackexchange.com/questions/242087/run-commands-in-parallel-and-wait-for-one-group-of-commands-to-finish-before-sta – Wissam Roujoulah Dec 01 '16 at 10:28
0

I'm not sure where you are getting your error message from. Normally the syntax would be

parallel -j 100% python3 -c main.py ::: input*

For example, if you just have 2 files inputa, inputb this runs in parallel

python3 -c main.py inputa
python3 -c main.py inputb

If you have 8 cpu cores, and, say, 10 input* files, the -j 100% argument makes parallel run only 8 commands at once at the start. When one finishes, the next command will be run until all 10 are done. We rely on normal Linux scheduling to assign each command to a separate cpu.

meuh
  • 51,383