Introduction to Control Groups (cgroups)
Cgroups is a Linux kernel feature to limit accounts and isolate the resources usage of process.
The cgroup model is one or more separate, unconnected trees of tasks and many different hierarchies of cgroups can exist simultaneously on a system. A single hierarchy can have one or more subsystems attached to it, but any single subsystem (such as cpu) cannot be attached to more than one hierarchy if one of those hierarchies has a different subsystem attached to it already.
Multiple separate hierarchies of cgroups are necessary because each hierarchy is attached to one or more subsystems. A subsystem (resource controller) represents a single resource such as CPU time or memory.
Each time a new hierarchy is created on the systems, all tasks on the system are initially members of the default cgroup of that hierarchy, which is known as the root cgroup.
Further a child task automatically inherits the cgroup membership of its parent but can be moved to different cgroups as needed.
You can find enabled control groups on your computer via proc filesystem:
cat /proc/cgroups or
ls -l /sys/fs/cgroup/
Linux kernel provides support for following control group subsystems:
cpuset - provide a mechanism for assigning a set of CPUs and Memory Nodes to a set of tasks.
cpu - uses the scheduler to provide cgroup tasks access to the processor resources.
cpuacct - account the CPU usage of these groups of tasks.
blkio - sets limits on input/output access to and from block devices such as physical drives (disk, solid state, or USB).
memory - isolates the memory behaviour of a group of tasks from the rest of the system.
devices - allows or denies access to devices by tasks in a cgroup.
freezer - start or stop sets of tasks or a task in order to schedule the resources of a machine.
net_cls - this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task.
net_prio - provides a dynamically way to set the priority of network traffic per network interface.
perf_event - provides access to perf events to a group.
hugetlb - allows to limit the HugeTLB (hugepagesize) usage per control group and enforces the controller limit during page fault.
pids - Used to allow a cgroup hierarchy to stop any new tasks from being fork()'d or clone()'d after a certain limit is reached.
systemd - create a cgroup for each service when starting it.
Example
Systemd with Cgroups
Through Systemd, we can limit resources for our test service
Create and compile a test c program that is using 100% of a single core and put it on /root folder
#include <stdio.h>
int main()
{
for (;;);
return 0;
}Create the following systemd service with limited cpu to 20% percentage and share to 200 (default is 1024 for each core) and memory limit to 50M
/etc/systemd/system/test.service
[Unit]
Description=Stress CPU
[Service]
ExecStart=/root/test
ExecStop=/bin/kill -WINCH ${MAINPID}
MemoryAccounting=true
CPUShares=200
CPUQuota=20%
MemoryLimit=50M
[Install]
WantedBy=multi-user.target
Start service:
systemctl start test.service
Run systemd-cgtop to see the usage of our test service.
The result should be something like below:
Path Tasks %CPU Memory Input/s Output/s
/system.slice/test.service 1 19.7 84.0K - -
To see the difference just remove from the service the CPUShares and CPUQuota to see that the CPU usage for a single core will be 100%:
Path Tasks %CPU Memory Input/s Output/s
/system.slice/test.service 1 99.4 88.0K - -
To change the values when the service is running and save this:
systemctl set-property test CPUShares=500 CPUQuota=50% MemoryLimit=100M
Docker with Cgroups
We can create a container that is running our test software and we can set the container to run with ‘cpu.shares’ to assign a relative CPU share and how many cores with cpuset.cpus.
Example
docker run -d \
--name='test_with_cgroup' \
--cpuset-cpus=0 \
--cpu-shares=50 \
/bin/bash /tmp/test
Again to see the difference just remove the lines CPUShares and CPUset to see that the CPU usage for a single core will be 100%
Limiting Users Resources with cgroups
Create config file /etc/cgconfig.conf
group test {
cpu {
cpu.shares="25";
}
cpuacct {
cpuacct.usage="0";
}
memory {
memory.limit_in_bytes="1G";
memory.memsw.limit_in_bytes="1G";
}
}
Create config file /etc/cgrules.conf
#<user/group> <controller(s)> <cgroup>
@test cpu,cpuacct,memory test
Run configuration parcer to parse and load the specified cgroups configuration file
cgconfigparser -l /etc/cgconfig.conf
Run the Rule engine daemon
cgrulesengd
Or you can use systemd to start and enable it:
systemctl enable cgconfig
systemctl start cgconfig
Sources
https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01