Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Parallelism, or parallelization, or parallel computing, means making use of multiple processors to complete a task more quickly than on a single processor.

For using multiple threads when the presence of more than one processor is irrelevant, use or instead. For parallelism over multiple machines, and not just over multiple processors in the same machine, use in addition to or instead of this tag.

is a common tool for parallel execution.

Useful links

277 questions
16
votes
5 answers

How to run scripts in parallel on a remote machine?

I can ssh into a remote machine that has 64 cores. Lets say I need to run 640 shell scripts in parallel on this machine. How do I do this? I can see splitting the 640 scripts into 64 groups each of 10 scripts. How would I then run each of these…
Tom
  • 263
1
vote
1 answer

Does parallel limit cause some items to not be checked?

I have near 400 git repos on my machine. And this is a script that I use to find their status collectively: function Check() { gitFolder="$1" parent=$(dirname $gitFolder); Status=$(git -C $parent status) if [[ $Status == *Changes*…
1
vote
1 answer

Could the situation: many CPUs simultaneously read the same single file, slow down the speed of reading of every CPU?

I am running cat to separately combine file_X with a numerous number of files e.g. file_1 to file_100000000000. Due to the large number, I distributed the job to a node with 64 CPUs to run parallelly on each CPU. Each job is ran in a sub-folder, and…
1
vote
1 answer

Running a for loop in parallel

I have a question regarding linux parallel scripts: Context: I need to process many .dat files Code I have so far: #!/bin/bash #PBS -S /bin/bash # set parallel parameters #PBS -r n #PBS -l walltime=2:00:00 #PBS -l procs=298 #PBS -l pmem=1600m…
Dan
  • 11
0
votes
1 answer

Download data in parallel

I have to download 20G of data using ftp .Can I do this in parallel(gnu?) Here are some of the links for the data. ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR030/ERR030893/ERR030893.fastq.gz …
Ron
  • 1,057