2

I found all processes on my machine to only run on a single core and their core affinity set to 0. Here is a small python script which reproduces this for me:

import multiprocessing
import numpy as np

def do_a_lot_of_compute(a):
    for i in range(1000):
        a = a * np.random.randn(123789)
    return a


if name == 'main':
    with multiprocessing.Pool() as pool:
        pool.map(do_a_lot_of_compute, np.arange(10000))

htop looks like this: enter image description here And the core affinities are:

pid 15977's current affinity list: 0
pid 15978's current affinity list: 0
pid 15979's current affinity list: 0
pid 15980's current affinity list: 0
pid 15981's current affinity list: 0
pid 15982's current affinity list: 0
pid 15983's current affinity list: 0
pid 15984's current affinity list: 0
pid 15985's current affinity list: 0

So my question boils down to: Why are the core affinities all set to 0? There are no OMP or KMP environment variables set.

Roofkiller
  • 21
  • 1
    What happens when you explictly tell your script to use certain cores? E.g. taskset $mask $script -- see here : https://baiweiblog.wordpress.com/2017/11/02/how-to-set-processor-affinity-in-linux-using-taskset/ and the taskset manual page. – schaiba Sep 13 '20 at 07:17
  • Manually overwriting the task affinities did not help. It seems that this issue was related to Slurm and PBS which both enforce the process affinity to a single core if one does not explicitly requests multiple cpu cores. – Roofkiller Sep 13 '20 at 08:36
  • If you have the answer please post it so others can benefit . – schaiba Sep 13 '20 at 09:17

1 Answers1

0

The problem was related to SLURM and PBS setting the core affinity based on the number of requested cores. In SLURM adding the following line enables the use of all cores:

#SBATCH --cpus-per-task=8
Roofkiller
  • 21