Andreas Christoforou         home     posts

Introduction to Control Groups (cgroups)

Cgroups is a Linux kernel feature to limit accounts and isolate the resources usage of process.

The cgroup model is one or more separate, unconnected trees of tasks and many different hierarchies of cgroups can exist simultaneously on a system. A single hierarchy can have one or more subsystems attached to it, but any single subsystem (such as cpu) cannot be attached to more than one hierarchy if one of those hierarchies has a different subsystem attached to it already.

Multiple separate hierarchies of cgroups are necessary because each hierarchy is attached to one or more subsystems. A subsystem (resource controller) represents a single resource such as CPU time or memory.

Each time a new hierarchy is created on the systems, all tasks on the system are initially members of the default cgroup of that hierarchy, which is known as the root cgroup.

Further a child task automatically inherits the cgroup membership of its parent but can be moved to different cgroups as needed.

You can find enabled control groups on your computer via proc filesystem:

    cat /proc/cgroups or
      
    ls -l /sys/fs/cgroup/

Linux kernel provides support for following control group subsystems:

cpuset -  provide a mechanism for assigning a set of CPUs and Memory Nodes to a set of tasks.
cpu - uses the scheduler to provide cgroup tasks access to the processor resources.
cpuacct - account the CPU usage of these groups of tasks.
blkio - sets limits on input/output access to and from block devices such as physical drives (disk, solid state, or USB).
memory - isolates the memory behaviour of a group of tasks from the rest of the system.
devices - allows or denies access to devices by tasks in a cgroup. 
freezer - start or stop sets of tasks or a task in order to schedule the resources of a machine.
net_cls - this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task. 
net_prio - provides a dynamically way to set the priority  of network traffic per network interface.
perf_event - provides access to perf events  to a group.
hugetlb - allows to limit the HugeTLB (hugepagesize) usage per control group and enforces the controller limit during page fault.
pids - Used to allow a cgroup hierarchy to stop any new tasks from being fork()'d or clone()'d after a certain limit is reached.
systemd - create a cgroup for each service when starting it.

Example

Systemd with Cgroups

Through Systemd, we can limit resources for our test service

Create and compile a test c program that is using 100% of a single core and put it on /root folder

#include <stdio.h>
int main()
{    
    for (;;);
    return 0;
}

Create the following systemd service with limited cpu to 20% percentage and share to 200 (default is 1024 for each core) and memory limit to 50M

/etc/systemd/system/test.service

[Unit]
Description=Stress CPU

[Service]
ExecStart=/root/test
ExecStop=/bin/kill -WINCH ${MAINPID}
MemoryAccounting=true 
CPUShares=200
CPUQuota=20%
MemoryLimit=50M

[Install]
WantedBy=multi-user.target

Start service:

systemctl start test.service

Run systemd-cgtop to see the usage of our test service.

The result should be something like below:

Path                      Tasks   %CPU   Memory  Input/s Output/s
/system.slice/test.service  1     19.7    84.0K     -        -

To see the difference just remove from the service the CPUShares and CPUQuota to see that the CPU usage for a single core will be 100%:

Path                      Tasks   %CPU   Memory  Input/s Output/s
/system.slice/test.service  1     99.4    88.0K     -        -

To change the values when the service is running and save this:

systemctl set-property test CPUShares=500 CPUQuota=50% MemoryLimit=100M

Docker with Cgroups

We can create a container that is running our test software and we can set the container to run with ‘cpu.shares’ to assign a relative CPU share and how many cores with cpuset.cpus.

Example

docker run -d \
--name='test_with_cgroup' \
--cpuset-cpus=0 \
--cpu-shares=50 \
/bin/bash /tmp/test

Again to see the difference just remove the lines CPUShares and CPUset to see that the CPU usage for a single core will be 100%

Limiting Users Resources with cgroups

Create config file /etc/cgconfig.conf

group test {
     	cpu {
		cpu.shares="25";
     	}
     	cpuacct {
            	cpuacct.usage="0";
     	}
     	memory {
            	memory.limit_in_bytes="1G";
            	memory.memsw.limit_in_bytes="1G";
     	}
}

Create config file /etc/cgrules.conf

#<user/group>         <controller(s)>         <cgroup>
@test                 cpu,cpuacct,memory        test

Run configuration parcer to parse and load the specified cgroups configuration file

cgconfigparser -l /etc/cgconfig.conf

Run the Rule engine daemon

cgrulesengd

Or you can use systemd to start and enable it:

systemctl enable cgconfig
systemctl start cgconfig

Sources

https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01