Chapter 2. Using Control Groups

The easiest way to work with cgroups is to install the libcgroup package, which contains a number of cgroup-related command line utilities and their associated man pages. It is possible to mount hierarchies and set cgroup parameters (non-persistently) using shell commands and utilities available on any system. However, using the libcgroup-provided utilities simplifies the process and extends your capabilities. Therefore, this guide focuses on libcgroup commands throughout. In most cases, we have included the equivalent shell commands to help describe the underlying mechanism. However, we recommend that you use the libcgroup commands wherever practical.

Installing the libcgroup package

In order to use cgroups, first ensure the libcgroup package is installed on your system by running, as root:

~]# yum install libcgroup

2.1. The cgconfig Service

The cgconfig service installed with the libcgroup package provides a convenient way to create hierarchies, attach subsystems to hierarchies, and manage cgroups within those hierarchies. It is recommended that you use cgconfig to manage hierarchies and cgroups on your system.

The cgconfig service is not started by default on Red Hat Enterprise Linux 6. When you start the service with chkconfig, it reads the cgroup configuration file - /etc/cgconfig.conf. Cgroups are therefore recreated from session to session and become persistent. Depending on the contents of the configuration file, cgconfig can create hierarchies, mount necessary file systems, create cgroups, and set subsystem parameters for each group.

The default /etc/cgconfig.conf file installed with the libcgroup package creates and mounts an individual hierarchy for each subsystem, and attaches the subsystems to these hierarchies.

If you stop the cgconfig service (with the service cgconfig stop command), it unmounts all the hierarchies that it mounted.

2.1.1. The /etc/cgconfig.conf File

The /etc/cgconfig.conf file contains two major types of entry - mount and group. Mount entries create and mount hierarchies as virtual file systems, and attach subsystems to those hierarchies. Mount entries are defined using the following syntax:

mount { <controller> = <path>; . . . . . . }

See Example 2.1, "Creating a mount entry" for an example usage.

Example 2.1. Creating a mount entry

The following example creates a hierarchy for the cpuset subsystem:

mount { cpuset = /cgroup/red;}

the equivalent of the shell commands:

~]# mkdir /cgroup/red~]# mount -t cgroup -o cpuset red /cgroup/red

Group entries create cgroups and set subsystem parameters. Group entries are defined using the following syntax:

group <name> { [<permissions>] <controller> { <param name> = <param value>; . . . . . . } . . . . . . }

Note that the permissions section is optional. To define permissions for a group entry, use the following syntax:

perm { task { uid = <task user>; gid = <task group>; } admin {   uid = <admin name>;   gid = <admin group>; }}

See Example 2.2, "Creating a group entry" for example usage:

Example 2.2. Creating a group entry

The following example creates a cgroup for SQL daemons, with permissions for users in the sqladmin group to add tasks to the cgroup and the root user to modify subsystem parameters:

group daemons/sql { perm { task { uid = root; gid = sqladmin; } admin { uid = root; gid = root; } } cpu { cpuset.mems = 0; cpuset.cpus = 0; }}

When combined with the example of the mount entry in Example 2.1, "Creating a mount entry", the equivalent shell commands are:

~]# mkdir -p /cgroup/red/daemons/sql~]# chown root:root /cgroup/red/daemons/sql/*~]# chown root:sqladmin /cgroup/red/daemons/sql/tasks~]# echo 0 > /cgroup/red/daemons/sql/cpuset.mems~]# echo 0 > /cgroup/red/daemons/sql/cpuset.cpus

Restart the cgconfig service for the changes to take effect

You must restart the cgconfig service for the changes in the /etc/cgconfig.conf to take effect. However, note that restarting this service causes the entire cgroup hierarchy to be rebuilt, which removes any previously existing cgroups (for example, any existing cgroups used by libvirtd). To restart the cgconfig service, use the following command:

~]# service cgconfig restart

When you install the libcgroup package, a sample configuration file is written to /etc/cgconfig.conf. The hash symbols ('#') at the start of each line comment that line out and make it invisible to the cgconfig service.

2.2. Creating a Hierarchy and Attaching Subsystems

Effects on running systems

The following instructions, which cover creating a new hierarchy and attaching subsystems to it, assume that cgroups are not already configured on your system. In this case, these instructions will not affect the operation of the system. Changing the tunable parameters in a cgroup with tasks, however, may immediately affect those tasks. This guide alerts you the first time it illustrates changing a tunable cgroup parameter that may affect one or more tasks.

On a system on which cgroups are already configured (either manually, or by the cgconfig service) these commands will fail unless you first unmount existing hierarchies, which will affect the operation of the system. Do not experiment with these instructions on production systems.

To create a hierarchy and attach subsystems to it, edit the mount section of the /etc/cgconfig.conf file as root. Entries in the mount section have the following format:

subsystem = /cgroup/hierarchy;

When cgconfig next starts, it will create the hierarchy and attach the subsystems to it.

The following example creates a hierarchy called cpu_and_mem and attaches the cpu, cpuset, cpuacct, and memory subsystems to it.

mount { cpuset  = /cgroup/cpu_and_mem; cpu = /cgroup/cpu_and_mem; cpuacct = /cgroup/cpu_and_mem; memory  = /cgroup/cpu_and_mem;}

Alternative method

You can also use shell commands and utilities to create hierarchies and attach subsystems to them.

Create a mount point for the hierarchy as root. Include the name of the cgroup in the mount point:

~]# mkdir /cgroup/name

For example:

~]# mkdir /cgroup/cpu_and_mem

Next, use the mount command to mount the hierarchy and simultaneously attach one or more subsystems. For example:

~]# mount -t cgroup -o subsystems name /cgroup/name

Where subsystems is a comma-separated list of subsystems and name is the name of the hierarchy. Brief descriptions of all available subsystems are listed in Available Subsystems in Red Hat Enterprise Linux, and Chapter 3, Subsystems and Tunable Parameters provides a detailed reference.

Example 2.3. Using the mount command to attach subsystems

In this example, a directory named /cgroup/cpu_and_mem already exists, which will serve as the mount point for the hierarchy that you create. Attach the cpu, cpuset and memory subsystems to a hierarchy named cpu_and_mem, and mount the cpu_and_mem hierarchy on /cgroup/cpu_and_mem:

~]# mount -t cgroup -o cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

You can list all available subsystems along with their current mount points (i.e. where the hierarchy they are attached to is mounted) with the lssubsys ^[3] command:

~]# lssubsys -amcpu,cpuset,memory /cgroup/cpu_and_memnet_clsnscpuacctdevicesfreezerblkio

This output indicates that:

the cpu, cpuset and memory subsystems are attached to a hierarchy mounted on /cgroup/cpu_and_mem, and
the net_cls, ns, cpuacct, devices, freezer and blkio subsystems are as yet unattached to any hierarchy, as illustrated by the lack of a corresponding mount point.

2.3. Attaching Subsystems to, and Detaching Them From, an Existing Hierarchy

To add a subsystem to an existing hierarchy, detach it from an existing hierarchy, or move it to a different hierarchy, edit the mount section of the /etc/cgconfig.conf file as root, using the same syntax described in Section 2.2, "Creating a Hierarchy and Attaching Subsystems". When cgconfig next starts, it will reorganize the subsystems according to the hierarchies that you specify.

Alternative method

To add an unattached subsystem to an existing hierarchy, remount the hierarchy. Include the extra subsystem in the mount command, together with the remount option.

Example 2.4. Remounting a hierarchy to add a subsystem

The lssubsys command shows cpu, cpuset, and memory subsystems attached to the cpu_and_mem hierarchy:

~]# lssubsys -amcpu,cpuset,memory /cgroup/cpu_and_memnet_clsnscpuacctdevicesfreezerblkio

Remount the cpu_and_mem hierarchy, using the remount option, and include cpuacct in the list of subsystems:

~]# mount -t cgroup -o remount,cpu,cpuset,cpuacct,memory cpu_and_mem /cgroup/cpu_and_mem

The lssubsys command now shows cpuacct attached to the cpu_and_mem hierarchy:

~]# lssubsys -amcpu,cpuacct,cpuset,memory /cgroup/cpu_and_memnet_clsnsdevicesfreezerblkio

Analogously, you can detach a subsystem from an existing hierarchy by remounting the hierarchy and omitting the subsystem name from the -o options. For example, to then detach the cpuacct subsystem, simply remount and omit it:

~]# mount -t cgroup -o remount,cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

2.4. Unmounting a Hierarchy

You can unmount a hierarchy of cgroups with the umount command:

~]# umount /cgroup/name

For example:

~]# umount /cgroup/cpu_and_mem

If the hierarchy is currently empty (that is, it contains only the root cgroup) the hierarchy is deactivated when it is unmounted. If the hierarchy contains any other cgroups, the hierarchy remains active in the kernel even though it is no longer mounted.

To remove a hierarchy, ensure that all child cgroups are removed before you unmount the hierarchy, or use the cgclear command which can deactivate a hierarchy even when it is not empty - refer to Section 2.12, "Unloading Control Groups".

2.5. Creating Control Groups

Use the cgcreate command to create cgroups. The syntax for cgcreate is:

cgcreate -t uid:gid -a uid:gid -g subsystems:path

where:

-t (optional) - specifies a user (by user ID, uid) and a group (by group ID, gid) to own the tasks pseudo-file for this cgroup. This user can add tasks to the cgroup.
Removing tasks
Note that the only way to remove a task from a cgroup is to move it to a different cgroup. To move a task, the user must have write access to the destination cgroup; write access to the source cgroup is unimportant.
-a (optional) - specifies a user (by user ID, uid) and a group (by group ID, gid) to own all pseudo-files other than tasks for this cgroup. This user can modify the access that the tasks in this cgroup have to system resources.
-g - specifies the hierarchy in which the cgroup should be created, as a comma-separated list of the subsystems associated with those hierarchies. If the subsystems in this list are in different hierarchies, the group is created in each of these hierarchies. The list of hierarchies is followed by a colon and the path to the child group relative to the hierarchy. Do not include the hierarchy mount point in the path.
For example, the cgroup located in the directory /cgroup/cpu_and_mem/lab1/ is called just lab1 - its path is already uniquely determined because there is at most one hierarchy for a given subsystem. Note also that the group is controlled by all the subsystems that exist in the hierarchies in which the cgroup is created, even though these subsystems have not been specified in the cgcreate command - refer to Example 2.5, "cgcreate usage".

Because all cgroups in the same hierarchy have the same controllers, the child group has the same controllers as its parent.

Example 2.5. cgcreate usage

Consider a system where the cpu and memory subsystems are mounted together in the cpu_and_mem hierarchy, and the net_cls controller is mounted in a separate hierarchy called net. Run the following command:

~]# cgcreate -g cpu,net_cls:/test-subgroup

The cgcreate command creates two groups named test-subgroup, one in the cpu_and_mem hierarchy and one in the net hierarchy. The test-subgroup group in the cpu_and_mem hierarchy is controlled by the memory subsystem, even though it was not specified in the cgcreate command.

Alternative method

To create a child of the cgroup directly, use the mkdir command:

~]# mkdir /cgroup/hierarchy/name/child_name

For example:

~]# mkdir /cgroup/cpuset/lab1/group1

2.6. Removing Control Groups

Remove cgroups with the cgdelete, which has a syntax similar to that of cgcreate. Run the following command:

cgdelete subsystems:path

where:

subsystems is a comma-separated list of subsystems.
path is the path to the cgroup relative to the root of the hierarchy.

For example:

~]# cgdelete cpu,net_cls:/test-subgroup

cgdelete can also recursively remove all subgroups with the option -r.

When you delete a cgroup, all its tasks move to its parent group.

2.7. Setting Parameters

Set subsystem parameters by running the cgset command from a user account with permission to modify the relevant cgroup. For example, if /cgroup/cpuset/group1 exists, specify the CPUs to which this group has access with the following command:

cpuset]# cgset -r cpuset.cpus=0-1 group1

The syntax for cgset is:

cgset -r parameter=value path_to_cgroup

where:

parameter is the parameter to be set, which corresponds to the file in the directory of the given cgroup
value is the value for the parameter
path_to_cgroup is the path to the cgroup relative to the root of the hierarchy. For example, to set the parameter of the root group (if /cgroup/cpuacct/ exists), run:
```
cpuacct]# cgset -r cpuacct.usage=0 /
```
Alternatively, because . is relative to the root group (that is, the root group itself) you could also run:
```
cpuacct]# cgset -r cpuacct.usage=0 .
```
Note, however, that / is the preferred syntax.
Setting parameters for the root group
Only a small number of parameters can be set for the root group (such as the cpuacct.usage parameter shown in the examples above). This is because a root group owns all of the existing resources, therefore, it would make no sense to limit all existing processes by defining certain parameters, for example the cpuset.cpu parameter.
To set the parameter of group1, which is a subgroup of the root group, run:
```
cpuacct]# cgset -r cpuacct.usage=0 group1
```
A trailing slash on the name of the group (for example, cpuacct.usage=0 group1/) is optional.

The values that you can set with cgset might depend on values set higher in a particular hierarchy. For example, if group1 is limited to use only CPU 0 on a system, you cannot set group1/subgroup1 to use CPUs 0 and 1, or to use only CPU 1.

You can also use cgset to copy the parameters of one cgroup into another, existing cgroup. For example:

~]# cgset --copy-from group1/ group2/

The syntax to copy parameters with cgset is:

cgset --copy-from path_to_source_cgroup path_to_target_cgroup

where:

path_to_source_cgroup is the path to the cgroup whose parameters are to be copied, relative to the root group of the hierarchy
path_to_target_cgroup is the path to the destination cgroup, relative to the root group of the hierarchy

Ensure that any mandatory parameters for the various subsystems are set before you copy parameters from one group to another, or the command will fail. For more information on mandatory parameters, refer to Mandatory parameters.

Alternative method

To set parameters in a cgroup directly, insert values into the relevant subsystem pseudo-file using the echo command. For example, this command inserts the value 0-1 into the cpuset.cpus pseudo-file of the cgroup group1:

~]# echo 0-1 > /cgroup/cpuset/group1/cpuset.cpus

With this value in place, the tasks in this cgroup are restricted to CPUs 0 and 1 on the system.

2.8. Moving a Process to a Control Group

Move a process into a cgroup by running the cgclassify command:

~]# cgclassify -g cpu,memory:group1 1701

The syntax for cgclassify is:

cgclassify -g subsystems:path_to_cgroup pidlist

where:

subsystems is a comma-separated list of subsystems, or * to launch the process in the hierarchies associated with all available subsystems. Note that if cgroups of the same name exist in multiple hierarchies, the -g option moves the processes in each of those groups. Ensure that the cgroup exists within each of the hierarchies whose subsystems you specify here.
path_to_cgroup is the path to the cgroup within its hierarchies
pidlist is a space-separated list of process identifier (PIDs)

You can also add the --sticky option before the pid to keep any child processes in the same cgroup. If you do not set this option and the cgred service is running, child processes will be allocated to cgroups based on the settings found in /etc/cgrules.conf. The process itself, however, will remain in the cgroup in which you started it.

Using cgclassify, you can move several processes simultaneously. For example, this command moves the processes with PIDs 1701 and 1138 into cgroup group1/:

~]# cgclassify -g cpu,memory:group1 1701 1138

Note that the PIDs to be moved are separated by spaces and that the groups specified should be in different hierarchies.

Alternative method

To move a process into a cgroup directly, write its PID to the tasks file of the cgroup. For example, to move a process with the PID 1701 into a cgroup at /cgroup/lab1/group1/:

~]# echo 1701 > /cgroup/lab1/group1/tasks

2.8.1. The cgred Service

Cgred is a service (which starts the cgrulesengd daemon) that moves tasks into cgroups according to parameters set in the /etc/cgrules.conf file. Entries in the /etc/cgrules.conf file can take one of the two forms:

user subsystems control_group
user:command subsystems control_group

For example:

mariadevices/usergroup/staff

This entry specifies that any processes that belong to the user named maria access the devices subsystem according to the parameters specified in the /usergroup/staff cgroup. To associate particular commands with particular cgroups, add the command parameter, as follows:

maria:ftpdevices/usergroup/staff/ftp

The entry now specifies that when the user named maria uses the ftp command, the process is automatically moved to the /usergroup/staff/ftp cgroup in the hierarchy that contains the devices subsystem. Note, however, that the daemon moves the process to the cgroup only after the appropriate condition is fulfilled. Therefore, the ftp process might run for a short time in the wrong group. Furthermore, if the process quickly spawns children while in the wrong group, these children might not be moved.

Entries in the /etc/cgrules.conf file can include the following extra notation:

@ - when prefixed to user, indicates a group instead of an individual user. For example, @admins are all users in the admins group.
* - represents "all". For example, * in the subsystem field represents all subsystems.
% - represents an item the same as the item in the line above. For example:
```
@adminstaffdevices/admingroup@labstaff%%
```

2.9. Starting a Process in a Control Group

Mandatory parameters

Some subsystems have mandatory parameters that must be set before you can move a task into a cgroup which uses any of those subsystems. For example, before you move a task into a cgroup which uses the cpuset subsystem, the cpuset.cpus and cpuset.mems parameters must be defined for that cgroup.

The examples in this section illustrate the correct syntax for the command, but only work on systems on which the relevant mandatory parameters have been set for any controllers used in the examples. If you have not already configured the relevant controllers, you cannot copy example commands directly from this section and expect them to work on your system.

Refer to Chapter 3, Subsystems and Tunable Parameters for a description of which parameters are mandatory for given subsystems.

Launch processes in a cgroup by running the cgexec command. For example, this command launches the lynx web browser within the group1 cgroup, subject to the limitations imposed on that group by the cpu subsystem:

~]# cgexec -g cpu:group1 lynx http://www.redhat.com

The syntax for cgexec is:

cgexec -g subsystems:path_to_cgroup command arguments

where:

subsystems is a comma-separated list of subsystems, or * to launch the process in the hierarchies associated with all available subsystems. Note that, as with cgset described in Section 2.7, "Setting Parameters", if cgroups of the same name exist in multiple hierarchies, the -g option creates processes in each of those groups. Ensure that the cgroup exists within each of the hierarchies whose subsystems you specify here.
path_to_cgroup is the path to the cgroup relative to the hierarchy.
command is the command to run.
arguments are any arguments for the command.

You can also add the --sticky option before the command to keep any child processes in the same cgroup. If you do not set this option and the cgred daemon is running, child processes will be allocated to cgroups based on the settings found in /etc/cgrules.conf. The process itself, however, will remain in the cgroup in which you started it.

Alternative method

When you start a new process, it inherits the group of its parent process. Therefore, an alternative method for starting a process in a particular cgroup is to move your shell process to that group (refer to Section 2.8, "Moving a Process to a Control Group"), and then launch the process from that shell. For example:

~]# echo $$ > /cgroup/lab1/group1/tasks~]# lynx

Note that after exiting lynx, your existing shell is still in the group1 cgroup. Therefore, an even better way would be:

~]# sh -c "echo \$$ > /cgroup/lab1/group1/tasks && lynx"

2.9.1. Starting a Service in a Control Group

You can start certain services in a cgroup. Services that can be started in cgroups must:

use a /etc/sysconfig/servicename file
use the daemon() function from /etc/init.d/functions to start the service

To make an eligible service start in a cgroup, edit its file in the /etc/sysconfig directory to include an entry in the form CGROUP_DAEMON="subsystem:control_group" where subsystem is a subsystem associated with a particular hierarchy, and control_group is a cgroup in that hierarchy. For example:

CGROUP_DAEMON="cpuset:daemons/sql"

2.9.2. Process Behavior in the Root Control Group

Certain blkio and cpu configuration options affect processes (tasks) running in the root cgroup in a different way than those in a subgroup. Consider the following example:

Create two subgroups under one root group: /rootgroup/red/ and /rootgroup/blue/
In each subgroup and in the root group, define the cpu.shares configuration option and set it to 1.

In the scenario configured above, one process placed in each group (that is, one task in /rootgroup/tasks, /rootgroup/red/tasks and /rootgroup/blue/tasks) ends up consuming 33.33% of the CPU:

/rootgroup/ process:  33.33%/rootgroup/blue/ process: 33.33%/rootgroup/red/ process:  33.33%

Any other processes placed in subgroups blue and red result in the 33.33% percent of the CPU assigned to that specific subgroup to be split among the multiple processes in that subgroup.

However, multiple processes placed in the root group cause the CPU resource to be split per process, rather than per group. For example, if /rootgroup/ contains three processes, /rootgroup/red/ contains one process and /rootgroup/blue/ contains one process, and the cpu.shares option is set to 1 in all groups, the CPU resource is divided as follows:

/rootgroup/ processes: 20% + 20% + 20%/rootgroup/blue/ process: 20%/rootgroup/red/ process:  20%

Therefore, it is recommended to move all processes from the root group to a specific subgroup when using the blkio and cpu configuration options which divide an available resource based on a weight or a share (for example, cpu.shares or blkio.weight). To move all tasks from the root group into a specific subgroup, you can use the following commands:

rootgroup]# cat tasks >> red/tasksrootgroup]# echo > tasks

2.10. Generating the /etc/cgconfig.conf File

Configuration for the /etc/cgconfig.conf file can be generated from the current cgroup configuration using the cgsnapshot utility. This utility takes a snapshot of the current state of all subsystems and their cgroups and returns their configuration as it would appear in the /etc/cgconfig.conf file. Example 2.6, "Using the cgsnapshot utility" shows an example usage of the cgsnapshot utility.

Example 2.6. Using the cgsnapshot utility

Configure cgroups on the system using the following commands:

~]# mkdir /cgroup/cpu~]# mount -t cgroup -o cpu cpu /cgroup/cpu~]# mkdir /cgroup/cpu/lab1~]# mkdir /cgroup/cpu/lab2~]# echo 2 > /cgroup/cpu/lab1/cpu.shares~]# echo 3 > /cgroup/cpu/lab2/cpu.shares~]# echo 5000000 > /cgroup/cpu/lab1/cpu.rt_period_us~]# echo 4000000 > /cgroup/cpu/lab1/cpu.rt_runtime_us~]# mkdir /cgroup/cpuacct~]# mount -t cgroup -o cpuacct cpuacct /cgroup/cpuacct

The above commands mounted two subsystems and created two cgroups, for the cpu subsystem, with specific values for some of their parameters. Executing the cgsnapshot command (with the -s option and an empty /etc/cgsnapshot_blacklist.conf file ^[4]) then produces the following output:

~]$ cgsnapshot -s# Configuration file generated by cgsnapshotmount { cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct;}group lab2 { cpu { cpu.rt_period_us="1000000"; cpu.rt_runtime_us="0"; cpu.shares="3"; }}group lab1 { cpu { cpu.rt_period_us="5000000"; cpu.rt_runtime_us="4000000"; cpu.shares="2"; }}

The -s option used in the example above tells cgsnapshot to ignore all warnings in the output file caused by parameters not being defined in the blacklist or whitelist of the cgsnapshot utility. For more information on parameter blacklisting, refer to Section 2.10.1, "Blacklisting Parameters". For more information on parameter whitelisting, refer to Section 2.10.2, "Whitelisting Parameters".

When not specifying any options, the output generated by cgsnapshot is returned on the standard output. Use the -f to specify a file to which the output should be redirected. For example:

~]$ cgsnapshot -f ~/test/cgconfig_test.conf

The -f option overwrites the specified file

When using the -f option, note that it overwrites any content in the file you specify. Therefore, it is recommended not to direct the output straight to the /etc/cgconfig.conf file.

The cgsnapshot utility can also create configuration files per subsystem. By specifying the name of a subsystem, the output will consist of the corresponding configuration for that subsystem:

~]$ cgsnapshot cpuacct# Configuration file generated by cgsnapshotmount { cpuacct = /cgroup/cpuacct;}

2.10.1. Blacklisting Parameters

The cgsnapshot utility allows parameter blacklisting. If a parameter is blacklisted, it does not appear in the output generated by cgsnapshot. By default, the /etc/cgsnapshot_blacklist.conf file is checked for blacklisted parameters. If a parameter is not present in the blacklist, the whitelist is checked. To specify a different blacklist, use the -b option. For example:

~]$ cgsnapshot -b ~/test/my_blacklist.conf

2.10.2. Whitelisting Parameters

The cgsnapshot utility also allows parameter whitelisting. If a parameter is whitelisted, it appears in the output generated by cgsnapshot. If a parameter is neither blacklisted or whitelisted, a warning appears informing of this:

~]$ cgsnapshot -f ~/test/cgconfig_test.confWARNING: variable cpu.rt_period_us is neither blacklisted nor whitelistedWARNING: variable cpu.rt_runtime_us is neither blacklisted nor whitelisted

By default, there is no whitelist configuration file. To specify which file to use as a whitelist, use the -w option. For example:

~]$ cgsnapshot -w ~/test/my_whitelist.conf

Specifying the -t option tells cgsnapshot to generate a configuration with parameters from the whitelist only.

2.11. Obtaining Information About Control Groups

2.11.1. Finding a Process

To find the cgroup to which a process belongs, run:

~]$ ps -O cgroup

Or, if you know the PID for the process, run:

~]$ cat /proc/PID/cgroup

2.11.2. Finding a Subsystem

To find the subsystems that are available in your kernel and how are they mounted together to hierarchies, run:

~]$ cat /proc/cgroups

Or, to find the mount points of particular subsystems, run:

~]$ lssubsys -m subsystems

where subsystems is a list of the subsystems in which you are interested. Note that the lssubsys -m command returns only the top-level mount point per each hierarchy.

2.11.3. Finding Hierarchies

It is recommended that you mount hierarchies under /cgroup. Assuming this is the case on your system, list or browse the contents of that directory to obtain a list of hierarchies. If tree is installed on your system, run it to obtain an overview of all hierarchies and the cgroups within them:

~]$ tree /cgroup

2.11.4. Finding Control Groups

To list the cgroups on a system, run:

~]$ lscgroup

You can restrict the output to a specific hierarchy by specifying a controller and path in the format controller:path. For example:

~]$ lscgroup cpuset:adminusers

lists only subgroups of the adminusers cgroup in the hierarchy to which the cpuset subsystem is attached.

2.11.5. Displaying Parameters of Control Groups

To display the parameters of specific cgroups, run:

~]$ cgget -r parameter list_of_cgroups

where parameter is a pseudo-file that contains values for a subsystem, and list_of_cgroups is a list of cgroups separated with spaces. For example:

~]$ cgget -r cpuset.cpus -r memory.limit_in_bytes lab1 lab2

displays the values of cpuset.cpus and memory.limit_in_bytes for cgroups lab1 and lab2.

If you do not know the names of the parameters themselves, use a command like:

~]$ cgget -g cpuset /

2.12. Unloading Control Groups

This command destroys all control groups

The cgclear command destroys all cgroups in all hierarchies. If you do not have these hierarchies stored in a configuration file, you will not be able to readily reconstruct them.

To clear an entire cgroup file system, use the cgclear command.

All tasks in the cgroup are reallocated to the root node of the hierarchies, all cgroups are removed, and the file system itself is unmounted from the system, thus destroying all previously mounted hierarchies. Finally, the directory where the cgroup file system was mounted is actually deleted.

Accurate listing of all mounted cgroups

Using the mount command to create cgroups (as opposed to creating them using the cgconfig service) results in the creation of an entry in the /etc/mtab file (the mounted file systems table). This change is also reflected into the /proc/mounts file. However, the unloading of cgroups with the cgclear command, along with other cgconfig commands, uses a direct kernel interface which does not reflect its changes into the /etc/mtab file and only writes the new information into the /proc/mounts file. Thus, after unloading cgroups with the cgclear command, the unmounted cgroups may still be visible in the /etc/mtab file, and, consequently, displayed when the mount command is executed. Refer to the /proc/mounts file for an accurate listing of all mounted cgroups.

2.13. Using the Notification API

The cgroups notification API allows user space applications to receive notifications about the changing status of a cgroup. Currently, the notification API only supports monitoring of the Out of Memory (OOM) control file: memory.oom_control. To create a notification handler, write a C program using the following instructions:

Using the eventfd() function, create a file descriptor for event notifications. For more information, refer to the eventfd(2) man page.
To monitor the memory.oom_control file, open it using the open() function. For more information, refer to the open(2) man page.
Use the write() function to write the following arguments to the cgroup.event_control file of the cgroup whose memory.oom_control file you are monitoring:
```
<event_file_descriptor> <OOM_control_file_descriptor>
```
where:
- event_file_descriptor is used to open the cgroup.event_control file,
- and OOM_control_file_descriptor is used to open the respective memory.oom_control file.
For more information on writing to a file, refer to the write(1) man page.

When the above program is started, it will be notified of any OOM situation in the cgroup it is monitoring. Note that OOM notifications only work in non-root cgroups.

For more information on the memory.oom_control tunable parameter, refer to Section 3.7, "memory". For more information on configuring notifications for OOM control, refer to Example 3.3, "OOM Control and Notifications".

2.14. Additional Resources

The definitive documentation for cgroup commands are the manual pages provided with the libcgroup package. The section numbers are specified in the list of man pages below.

The libcgroup Man Pages

man 1 cgclassify - the cgclassify command is used to move running tasks to one or more cgroups.
man 1 cgclear - the cgclear command is used to delete all cgroups in a hierarchy.
man 5 cgconfig.conf - cgroups are defined in the cgconfig.conf file.
man 8 cgconfigparser - the cgconfigparser command parses the cgconfig.conf file and mounts hierarchies.
man 1 cgcreate - the cgcreate command creates new cgroups in hierarchies.
man 1 cgdelete - the cgdelete command removes specified cgroups.
man 1 cgexec - the cgexec command runs tasks in specified cgroups.
man 1 cgget - the cgget command displays cgroup parameters.
man 1 cgsnapshot - the cgsnapshot command generates a configuration file from existing subsystems.
man 5 cgred.conf - cgred.conf is the configuration file for the cgred service.
man 5 cgrules.conf - cgrules.conf contains the rules used for determining when tasks belong to certain cgroups.
man 8 cgrulesengd - the cgrulesengd service distributes tasks to cgroups.
man 1 cgset - the cgset command sets parameters for a cgroup.
man 1 lscgroup - the lscgroup command lists the cgroups in a hierarchy.
man 1 lssubsys - the lssubsys command lists the hierarchies containing the specified subsystems.

^[3]The lssubsys command is one of the utilities provided by the libcgroup package. You must install libcgroup to use it: refer to Chapter 2, Using Control Groups if you are unable to run lssubsys.

^[4]The cpu.shares parameter is specified in the /etc/cgsnapshot_blacklist.conf file by default, which would cause it to be omitted in the generated output in Example 2.6, "Using the cgsnapshot utility". Thus, for the purposes of the example, an empty /etc/cgsnapshot_blacklist.conf file is used.

Resource Management Guide

Chapter 2. Using Control Groups

2.1. The cgconfig Service

2.1.1. The /etc/cgconfig.conf File

2.2. Creating a Hierarchy and Attaching Subsystems

Alternative method

2.3. Attaching Subsystems to, and Detaching Them From, an Existing Hierarchy

Alternative method

2.4. Unmounting a Hierarchy

2.5. Creating Control Groups

Alternative method

2.6. Removing Control Groups

2.7. Setting Parameters

Alternative method

2.8. Moving a Process to a Control Group

Alternative method

2.8.1. The cgred Service

2.9. Starting a Process in a Control Group

Alternative method

2.9.1. Starting a Service in a Control Group

2.9.2. Process Behavior in the Root Control Group

2.10. Generating the /etc/cgconfig.conf File

2.10.1. Blacklisting Parameters

2.10.2. Whitelisting Parameters

2.11. Obtaining Information About Control Groups

2.11.1. Finding a Process

2.11.2. Finding a Subsystem

2.11.3. Finding Hierarchies

2.11.4. Finding Control Groups

2.11.5. Displaying Parameters of Control Groups

2.12. Unloading Control Groups

2.13. Using the Notification API

2.14. Additional Resources