How to ensure exclusive CPU availability for a running process?

Question

At first, the question seems to be a little bit silly/confusing as the OS does the job of managing process execution.

However, I want to measure how much some processes are CPU/IO-bound and I feel like my OS is interfering on my experiments with, for instance, scheduled OS processes.

Take as an example the following situation: I ran the process A twice and got the following output from the tool "time" (time columns in seconds):

+---+-------+---------+-----------+---------+
|Run|Process|User Time|System Time|Wall time|
+---+-------+---------+-----------+---------+
|1  |A      |196.3    |5.12       |148.86   |
|2  |A      |190.79   |4.93       |475.46   |
+---+-------+---------+-----------+---------+

As we can see, although the user and sys time are similar, the elapsed time of both drastically changes (diff. of ~5 min). Feels like something in my environment caused some sort of contention.

I want to stop every possible background process/services to avoid any kind of noise during my experiments but I consider myself a novice/intermediate unix-user and I don't know how to guarantee this.

I'm using Linux 4.4.0-45-generic with Ubuntu 14.04 LTS 64 bit.

I really appreciate the assistance. If you guys need any missing information, I will promptly edit my post.

CPU Info

$ grep proc /proc/cpuinfo | wc -l
8
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               4002.609
BogoMIPS:              7183.60
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

You can start your process using a hight priority nice -19 process — Dababi, Nov 28 '16 at 14:21

Rui F Ribeiro · Accepted Answer · 2019-07-11T09:55:11.377

You have a kernel option configuration where a CPU won't be used by the OS, it is called isolcpus.

isolcpus — Isolate CPUs from the kernel scheduler.

Synopsis isolcpus= cpu_number [, cpu_number ,...]

Description Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms. The only way to move a process onto or off an "isolated" CPU is via the CPU affinity syscalls. cpu_number begins at 0, so the maximum value is 1 less than the number of CPUs on the system.

This configuration I am about to describe how to setup, can have far more uses than for testing.

Meru for instance, uses this technology in their Linux-based AP controllers, to keep the network traffic from interfering with the inner workings of the OS, namely I/O operations.

I also use it in a very busy web frontend, for quite the same reasons: I have found out from life experience that I lost control too regularly for my taste of that server ; had to reboot it forcefully until I separated the front end daemon on it´s own dedicated CPUs.

As you have 8 CPUs, that you can check with the output of the command:

$ grep -c proc /proc/cpuinfo
8

or

$ lscpu | grep '^CPU.s'
CPU(s):                8

Add in Debian/Ubuntu in the file /etc/default/grub to the option GRUB_CMDLINE_LINUX:

GRUB_CMDLINE_LINUX="isolcpus=7"

(it is 7, because it starts in 0, and you have 8 cores)

Then run,

sudo update-grub

This is telling the kernel to not use one of your cores.

Reboot the system.

Then start your process.

Immediately after starting it, you can change for the 8th CPU (7 because 0 is the 1st), and be quite sure you are the only one using that CPU.

For that, use the command:

taskset -cp 7 PID_number

taskset - retrieve or set a processes’s CPU affinity

SYNOPSIS
   taskset [options] [mask | list ] [pid | command [arg]...]
DESCRIPTION

taskset is used to set or retrieve the CPU affinity of a running pro cess given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

For reading more about it, see: isolcpus, numactl and taskset

Also using ps -eF you should see in the PSR column the processor being used.

I have a server with CPU 2 and 3 isolated, and indeed, it can be seen with ps -e the only process in userland as intended, is pound.

# ps -eo psr,command | tr -s " " | grep "^ [2|3]"
 2 [cpuhp/2]
 2 [watchdog/2]
 2 [migration/2]
 2 [ksoftirqd/2]
 2 [kworker/2:0]
 2 [kworker/2:0H]
 3 [cpuhp/3]
 3 [watchdog/3]
 3 [migration/3]
 3 [ksoftirqd/3]
 3 [kworker/3:0]
 3 [kworker/3:0H]
 2 [kworker/2:1]
 3 [kworker/3:1]
 3 [kworker/3:1H]
 3 /usr/sbin/pound

If you compare it with the non-isolated CPUs, they are running many more things (the window below slides):

# ps -eo psr,command | tr -s " " | grep "^ [0|1]"
 0 init [2]
 0 [kthreadd]
 0 [ksoftirqd/0]
 0 [kworker/0:0H]
 0 [rcu_sched]
 0 [rcu_bh]
 0 [migration/0]
 0 [lru-add-drain]
 0 [watchdog/0]
 0 [cpuhp/0]
 1 [cpuhp/1]
 1 [watchdog/1]
 1 [migration/1]
 1 [ksoftirqd/1]
 1 [kworker/1:0]
 1 [kworker/1:0H]
 1 [kdevtmpfs]
 0 [netns]
 0 [khungtaskd]
 0 [oom_reaper]
 1 [writeback]
 0 [kcompactd0]
 0 [ksmd]
 1 [khugepaged]
 0 [crypto]
 1 [kintegrityd]
 0 [bioset]
 1 [kblockd]
 1 [devfreq_wq]
 0 [watchdogd]
 0 [kswapd0]
 0 [vmstat]
 1 [kthrotld]
 0 [kworker/0:1]
 0 [deferwq]
 0 [scsi_eh_0]
 0 [scsi_tmf_0]
 1 [vmw_pvscsi_wq_0]
 0 [bioset]
 1 [jbd2/sda1-8]
 1 [ext4-rsv-conver]
 0 [kworker/0:1H]
 1 [kworker/1:1H]
 1 [bioset]
 0 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 0 [jbd2/sda3-8]
 1 [ext4-rsv-conver]
 1 /usr/sbin/rsyslogd
 0 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
 1 /usr/sbin/cron
 0 /usr/sbin/sshd
 1 /usr/sbin/snmpd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid
 1 /sbin/getty 38400 tty1
 1 /lib/systemd/systemd-udevd --daemon
 0 /usr/sbin/xinetd -pidfile /run/xinetd.pid -stayalive
 1 [kworker/1:2]
 0 [kworker/u128:1]
 0 [kworker/0:2]
 0 [bioset]
 1 [xfsalloc]
 1 [xfs_mru_cache]
 1 [jfsIO]
 1 [jfsCommit]
 0 [jfsCommit]
 0 [jfsCommit]
 0 [jfsCommit]
 0 [jfsSync]
 1 [bioset]
 0 /usr/bin/monit -c /etc/monit/monitrc
 1 /usr/sbin/pound
 0 sshd: rui [priv]
 0 sshd: rui@pts/0,pts/1
 1 -bash
 1 -bash
 1 -bash
 1 [kworker/u128:0]
 1 -bash
 0 sudo su
 1 su
 1 bash
 0 bash
 0 logger -t cmdline root[/home/rui] 
 1 ps -eo psr,command
 0 tr -s 
 0 grep ^ [0|1]
 0 /usr/bin/vmtoolsd

I didn't know it was possible to do that :) I'm gonna check your references. Looking forward to mark this thread as resolved ;) — Jeanderson Candido, Nov 28 '16 at 14:47
I almost forgot... is there any way to monitor the execution to check if this isolation works? — Jeanderson Candido, Nov 28 '16 at 14:59
You are welcome. Added a small intro as the 2nd paragraph. Note that I have seen some tutorial in the net talking about taskset, but often they do not mention together with it isolcpus — Rui F Ribeiro, Nov 29 '16 at 02:07
FYI, four years later, isolcpus is now deprecated. That link recommends using cgroups to do it, which is what containers are based on. — Aaron D. Marasco, Dec 15 '21 at 23:53
my system regularly hangs when I run Machine Learning/Compute/Memory intensive workloads. Do you think its a good idea to reserve one core so that I can use it to open terminal when system hangs and kill processes? — Anmol Deep, Jan 24 '24 at 07:42
@AnmolDeep I had that need for this very same reason, OS handing. Whilst isolcpus still work, see also the previous comment/link about cgroups. — Rui F Ribeiro, Jan 24 '24 at 21:58

score 5 · Answer 2 · answered Mar 01 '22 at 15:57

5

as mentioned by @Aaron D. Marasco using cset from cpuset package is also possible. (isolcpus is not really deprecated.) One advantage of cset is no reboot between configs and the scheduler load balancer remains active if multiple cores are used.

reserve processor 5 and 11 for our tasks

cset shield --cpu 5,11 --kthread on # cpu 5 and 11, move kernel processes if possible

check what is running on our cpuset

cset shield -s -v

execute a task on the cpuset

cset shield -e bash

attach a process

cset shield --shield --pid 22010

detach a process

cset shield --unshield --pid 22010

to restore everything

cset shield --reset

answered Mar 01 '22 at 15:57

Matoran

51

This is very interesting. Though it appears as if you can have at most a single shield at once? So it seems like the case where you have two tasks dedicated to two different CPUs is not possible with cset. Or am I missing something – sherrellbc Nov 23 '23 at 17:28
@sherrellbc cset-shield is a tool for easy setup. If you want multiple sets you can use cset-set to create sets and cset-proc to set procs on that set. – Matoran Nov 24 '23 at 10:21

How to ensure exclusive CPU availability for a running process?

2 Answers2

Linked

Related