#+TITLE: Support for CPU Resource Constraints Set via Linux Control Groups

* Introduction

oneTBB respects constrains applied to the process. In the most common scenario, the library checks
CPU affinity mask to compute the number of worker threads it will instantiate by default. The same
mechanism is used to calculate the maximum concurrency level of a task arena or the number of slots
within it. This is also true when task arena is initialized with an instance of
=tbb::task_arena::constraints= whose settings of desired NUMA node, core type, and the others define
the CPU affinity mask, hence the maximum concurrency of the task arena.

However, there are cases when the limiting factor may come not only from the process affinity mask
but also from the CPU time quota applied to the process. CPU time quota allows more granular
management of resources than allocating specific portion of hardware threads to the process, and is
extensively used in containerized environments [1-3].

In Linux such CPU quota can be set through the control groups (aka cgroups) kernel feature, using
the CPU controller. The controller allows setting a =period=, which signifies the guaranteed amount
of time the process will be given when the scheduler allocates CPU time to it. Starting from Linux
3.2, this possibility was extended to provide CPU "bandwidth" control, which allows to define an
upper limit on the CPU time allocated to the process in a cgroup [4]. Within each given =period=
(microseconds), a task group is allocated up to =quota= microseconds of CPU time. That quota is
assigned to per-cpu run queues in slices as threads in the cgroup become runnable. Once all quota
has been assigned any additional requests for quota will result in those threads being throttled.
Throttled threads will not be able to run again until the next period when the quota is replenished
[5].

Therefore, simultaneously running more threads than the $quota / period$ will make additional
requests for allocation of CPU time to these threads, which will not be satisfied due to limitations
imposed on the process. As a result, the process will experience symptoms similar to those when
oversubscribing the system. Limiting the number of threads to the value from the formula above
avoids oversubscribing the platform and restores the performance. See [[speedup-chart.png][the chart]].

#+CAPTION: Speedup over sequential run of Pi oneTBB example when CPU resources are limited
#+NAME: speedup-chart.png
#+ATTR_HTML: :align center :width 800px
[[./speedup-chart.png]]

The interface for kernel's cgroups is provided through a pseudo-filesystem called cgroupfs. There
are two versions of cgroup which differ particularly in the formats of the files and their paths in
the pseudo-filesystem. For example, the CPU controller =quota= and =period= for the cgroup v1
interface can be found in =cpu.cfs_quota_us= and =cpu.cfs_period_us= files, while for the cgroup v2
they are separated by a whitespace and written in =cpu.max= file. The value of =-1= for the quota in
the cgroup v1 or =max= for the cgroup v2 indicates that the processes within such a group do not
have any bandwidth restriction in place.

*Note*: Cgroups allow setting usage limitations for various platform resources, including the
affinity mask which is set by the =cpuset= controller. The processes in cgroup inherit the affinity
mask. So, they are already respected by the library and does not require additional handling.

Although more and more Linux-based distributions of operating systems support cgroup version two,
there are relevant OSes that implement only the first version of the cgroup interface. In addition,
modern Linux kernels allow to use different cgroup versions simultaneously, by connecting various
controllers that exist in both versions to either one or the other cgroup interface. This makes it
important to support both cgroup versions.

* Proposal

Support CPU resource constraints imposed on the process through Linux control groups kernel feature.

This will affect the default number of threads instantiated by the library as well as the default
value for maximum concurrency of a =tbb::task_arena= instance disregarding whether it is
instantiated with =tbb::task_arena::constraints= specified or not. In all of these cases, the
obtained $quota / period$ ratio will be considered as another limiting factor in addition to the
affinity mask currently used. The resulting default value will be taken as a minimum between the
obtained value and the one determined previously.

The $quota / period$ value is rounded up because of the following considerations:
- It is likely the feature is to be mostly used in containerized environments, which provide a
  service rather than a single application. In most of the cases, containers sub-divide the platform
  by whole CPUs. Therefore, the division is going to be without a remainder.
- Likely the cgroup constraints will impose limitations on a group of processes. Since CPU resources
  are shared among all the constrained processes, TBB workers will likely be throttled even if ratio
  is rounded down.
- To avoid undersubscribing by refraining from utilizing full CPU bandwidth.

** Implementation Details

Since the cgroup interface is made through a pseudo-filesystem, to properly determine paths to the
files describing CPU quota and period, a =/proc/self/mounts= file can be parsed [6]. The format of
this file is documented in [7]. There are also routines that make possible not parsing that file
manually [8].

In addition to the mount point found after inspecting the =/proc/self/mounts= file, it is necessary
to look at the =/proc/self/cgroup= file to determine /relative path/ of the CPU controller in the
control group to which the process belongs. This path is relative to the mount point of the cgroup
hierarchy.

However, there are cases when the final path made up from concatenation of mount point and relative
path is not available in the containerized environment but exists on the host. In practice, however,
the files with identical content were found directly in the mount point folder on the container.
Thus, to make the algorithm more robust, it is necessary to try looking into both paths: a mount
point path and the concatenated one. On the other hand, the algorithm should not try investigating
all of the possible options and stop once the data is found and successfully read.

#+begin_src bash
  # Below lines use ${container_hash} for brevity, as a substitution for the actual container hash,
  # which equals to '3f54303c5a7e9b3fe9dc7206f6f030fe3b2956db0a1029d1906d742bcc03a7e0' in this example

  # Running in a Docker container.
  root@3f54303c5a7e:/# grep -E "cpu[, ].*cgroup" /proc/self/mounts
  cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0

  root@3f54303c5a7e:/# grep -E "cpu[, ].*:" /proc/self/cgroup
  3:cpu,cpuacct:/docker/${container_hash}

  root@3f54303c5a7e:/# ls /sys/fs/cgroup/cpu,cpuacct/docker/${container_hash}
  ls: cannot access '/sys/fs/cgroup/cpu,cpuacct/docker/${container_hash}': No such file or directory

  # Running in a host system.
  $ ls /sys/fs/cgroup/cpu,cpuacct/docker/${container_hash} | grep "period\|quota"
  cpu.cfs_period_us
  cpu.cfs_quota_us

  # Look only at the mount point found inside a Docker container.
  root@3f54303c5a7e:/# ls $(grep -E "cpu[, ].*cgroup" /proc/self/mounts | cut -d" " -f2) \
      | grep "period\|quota"
  cpu.cfs_period_us
  cpu.cfs_quota_us
#+end_src

*** Determining If Scheduling Constraints Are Set
As an optimization, it makes sense to first check whether any CPU scheduling constraints are set.
This can be achieved by parsing the =/proc/self/cgroup= file to identify entries with either the
CPU controller for cgroup v1 or any entry associated with cgroup v2.
Entries in this file are formatted as follows [4]:

=hierarchy-ID:controller-list:relative-path=

For cgroup v1, it is enough to find an entry with =cpu= in the controller-list field. For cgroup v2,
this is more complex. The controller-list field is always empty for cgroup v2, and the hierarchy ID
is always =0=. For example: =0::/cgroup-relative-path=. The only quick way to determine whether the
=cpu.max= setting is absent is by checking if the process uses the root cgroup (=/= relative path)
or not. If it does, then =cpu.max= cannot be set, as it is only possible to set it for non-root
cgroups [9]. However, the process can run under a cgroup namespace, where the "virtualized" root
cgroup may differ from the host's root cgroup [10].

So, for cgroup v2, we additionally need to check if the process is running under a
cgroup namespace by inspecting the cgroup of PID 1. On Linux, PID 1 corresponds to the init process.
For distributions using =systemd= (which constitutes the majority of modern Linux-based OSes),
=/proc/1/cgroup= would typically contain =0::/init.scope= [11]. Therefore, for systemd-based
distributions, it is sufficient to check whether the relative path in =/proc/1/cgroup= differs from
=/init.scope=. If it does, the process is running under a cgroup namespace, and it is possible to
apply CPU scheduling constraints.

*** Minimizing File Reads
On many modern systems, controllers are automatically mounted under =/sys/fs/cgroup= (v1
controllers) and =/sys/fs/cgroup/unified= (v2 controllers) [4]. In practice, v2 controllers usually
automatically mounted under v1 controllers path, that is =/sys/fs/cgroup=. So, to avoid parsing
=/proc/self/mounts=, these paths can be tried first.

Since each of the v1 controllers has an associated configuration option that must be set in order to
employ that controller [4], it makes sense to filter only those mount points that contain =cpu=
controller in the mount options.

*** The Algorithm Outline
Based on the described considerations, the algorithm outline should be the following:

1. Parse =/proc/self/cgroup= to:
   - Find relative paths to cgroup controllers
   - Determine whether CPU scheduling constraints can be set by:
     - Looking for =cpu= controller for cgroup v1
     - For cgroup v2, checking if the process is not running under root cgroup (with PID 1 check)
2. Apply found relative paths to the most probable paths and look for the information there:
   - =/sys/fs/cgroup= for cgroup v1 and v2
   - =/sys/fs/cgroup/unified= for cgroup v2
3. If not found, loop over =/proc/self/mounts= and find a mount point for cgroupfs
   - For cgroup v1, make sure the mount options contain =cpu= controller
4. Concatenate the found mount path with the path from (1) and look for the information there
5. Repeat from (3) until the info is found or the mount points are exhausted.

* References:

1. https://docs.docker.com/engine/containers/resource_constraints/#cpu
2. https://github.com/uxlfoundation/oneTBB/issues/190
3. https://github.com/uxlfoundation/oneTBB/issues/1760
4. https://man7.org/linux/man-pages/man7/cgroups.7.html
5. https://www.kernel.org/doc/html/v6.15/scheduler/sched-bwc.html
6. https://www.man7.org/linux/man-pages/man5/proc_mounts.5.html
7. https://www.man7.org/linux/man-pages/man5/fstab.5.html
8. https://www.man7.org/linux/man-pages/man3/getmntent.3.html
9. https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
10. https://www.man7.org/linux/man-pages/man7/cgroup_namespaces.7.html
11. https://www.man7.org/linux/man-pages/man7/systemd.special.7.html
